From owner-svn-src-all@freebsd.org Tue Nov 7 21:17:12 2017 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35AA6E650FA for ; Tue, 7 Nov 2017 21:17:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF2DA6C1AE for ; Tue, 7 Nov 2017 21:17:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x236.google.com with SMTP id 143so371987itf.1 for ; Tue, 07 Nov 2017 13:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=u3HMHM5MxxXNR4U1eD9bUlIqhNYcZOiowb8usUjYnUY=; b=Ndo1Yp6h66ynIW19AbPlIAK7R1t9KzaG6f2HrBMjS94eGRerZdeJUwdPcUv5n+h8SG I+kP2Y1TppsqdrstY3gvzWDCYfTNHonoUGkknCWWbkYbYmEdHm3OxonkwPC2umY6BS0l yv/m47C0PS8UlsVQuuKTZ6Nlv3MBaDaYyYhM8B3dw/CEzljDQLfhnplMGtIpF1hUKl3W 16VK9M69xKyy70IVhPfRBSb6egE8LrBDc8KGxkjUYu3GmSsOPnZp9t5l4FO2IJT73EIe ZtCWhlEYBeWingv6kYPbA7fIU7wtR3FAgmkEE3n/khrHnY2q5PlDYLy8SBwLr/s9hATV bh0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=u3HMHM5MxxXNR4U1eD9bUlIqhNYcZOiowb8usUjYnUY=; b=OV49vKCREZ5uSVI35Nv0SDkJX3zPscgD0rlhl9+fjifsEaw76gTUjCC1CE5BEfWakM NzhgHaZaoeyfkvN+CvhQzTkmP8WyJt/9zS6rYDYnakdMpEUWIy44oKHqOZyRqg+DquKr imzhyD6jD+WjtWg4Tbo8aOFSV07sU2dyh5ylcCMxqTycq4Ux9efXEmkyQe2szwYOZJIQ hX/5Pum1tO6cHPk9xYJJMWuCImXxtG350oSexa5KGQ6Po+bj5dWpkoLzNssL4hEFoZW3 QmJG9wzKKqNtmATvzjvuPi49oas2fmmkjDVUeYnzrf5pcs3pjN6DGX0ddHobod9tkWyf 8rEA== X-Gm-Message-State: AJaThX5SRVW3KRTyO3usCnPM65jpikSwXattm9PPwZnI1pMRBVxTEuNu WryQw7886ppaQ3OcqZl6TlZ8fl/GJOQ8ZUQynUQVSA== X-Google-Smtp-Source: ABhQp+QZVVnU5Q3OIkY5n7IOSJF2inxprBNhtBOlqwGtJT4wapbRarVWKsYF82P+j4nz6Y/nt2+Bsg30zPNlr5yMQUY= X-Received: by 10.36.101.207 with SMTP id u198mr718603itb.50.1510089431153; Tue, 07 Nov 2017 13:17:11 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.57.22 with HTTP; Tue, 7 Nov 2017 13:17:10 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:a004:68c9:b567:b3a8] In-Reply-To: <20171107174431.GQ2566@kib.kiev.ua> References: <201711070929.vA79TFTc096109@repo.freebsd.org> <2707886.I9a5Fez8At@ralph.baldwin.cx> <20171107173926.GP2566@kib.kiev.ua> <20171107174431.GQ2566@kib.kiev.ua> From: Warner Losh Date: Tue, 7 Nov 2017 14:17:10 -0700 X-Google-Sender-Auth: ElEoYQSibcasKuJxhlQ433Ob2Mk Message-ID: Subject: Re: svn commit: r325506 - in head: sbin/ifconfig sys/net sys/sys To: Konstantin Belousov Cc: John Baldwin , src-committers , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Nov 2017 21:17:12 -0000 On Tue, Nov 7, 2017 at 10:44 AM, Konstantin Belousov wrote: > On Tue, Nov 07, 2017 at 07:39:26PM +0200, Konstantin Belousov wrote: > > On Tue, Nov 07, 2017 at 09:06:52AM -0800, John Baldwin wrote: > > > On Tuesday, November 07, 2017 09:29:15 AM Konstantin Belousov wrote: > > > > Author: kib > > > > Date: Tue Nov 7 09:29:14 2017 > > > > New Revision: 325506 > > > > URL: https://svnweb.freebsd.org/changeset/base/325506 > > > > > > > > Log: > > > > Add a place for a driver to report rx timestamps in nanoseconds > from > > > > boot for the received packets. > > > > > > > > The rcv_tstmp field overlaps the place of Ln header length > indicators, > > > > not used by received packets. The basic pkthdr rearrangement > change > > > > in sys/mbuf.h was provided by gallatin. > > > > > > > > There are two accompanying M_ flags: M_TSTMP means that there is > the > > > > timestamp (and it was generated by hardware). > > > > > > > > Another flag M_TSTMP_HPREC indicates that the timestamp is > > > > high-precision. Practically M_TSTMP_HPREC means that hardware > > > > provided additional precision comparing with the stamps when the > flag > > > > is not set. E.g., for ConnectX all packets are stamped by hardware > > > > when PCIe transaction to write out the completion descriptor is > > > > performed, but PTP packet are stamped on port. For Intel cards, > when > > > > PTP assist is enabled, only PTP packets are stamped in the limited > > > > number of registers, so if Intel cards ever start support this > > > > mechanism, they would always set M_TSTMP | M_TSTMP_HPREC if > hardware > > > > timestamp is present for the given packet. > > > > > > > > Add IFCAP_HWRXTSTMP interface capability to indicate the support > for > > > > hardware rx timestamping, and ifconfig(8) command to toggle it. > > > > > > Hmm, other NICs (Chelsio T4 and later for example) support timestamps > that > > > aren't in nanoseconds but some other frequency (which are themselves > useful). > > > It would be nice to have a more flexible interface that supports not > only ns > > > timestamps. Perhaps a way to expose a direct hardware timestamp as a > > > "number" without a specific frequency? > > > > ConnectX does not provide ns-clocked counter either. It is some internal > > clock driven by a cristal with > 100MHz frequency. > > > > There is no much space in the pkthdr, and the request to provide the > > timestamp was in the context where the wall clock or some closely related > > timer is needed. Of course, I can put raw hardware timestamp into the > > packet header, but only instead of the reduced value. Then the consumer > > of the timestamp would need to find the interface which received the > > packet and call its method to convert ? We have only one consumer in > > tree (SO_TIMESTAMP) and perhaps one possible another consumer (TCP) for > > this data, both of which require wall clock, so would need to call into > > the method. > > > > Also please see the discussion in the referenced review about accuracy of > > the convertion. > > > > Important example are Intel cards where is only limited number of > > latched registers, and only PtP packets are stamped. This (and some > > quirk in ConnectX) explains the high-precision flag. > > > > And another consideration which was one of the strong argument for me > when I thought about this stuff: the convertion of the hardware timestamp > to the useful clock stamp depends on the clock calibraton data which might > not be available long time after the packet receive. In other words, when > the consumer would call into the interface method to convert raw timestamp, > it might be already not convertable (in kern_tc.c terms, timehands were > switched by tc_windup()). > On the other hand, if you are trying to steer or calibrate the clock in userland, doing the conversions may be losing data if the clocks are fast enough... But that was one of the big lessons I learned when doing timing gear: raw data is better than cooked data in userland, even if you instantly cook it in the same way you would have cooked it in the kernel before publishing. Given the quality of these clocks, though, I doubt it matters much at all... Warner