Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 14 Mar 2014 21:47:32 -0700
From:      Adrian Chadd <adrian@freebsd.org>
To:        Robert Watson <rwatson@freebsd.org>
Cc:        svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org
Subject:   Re: svn commit: r263198 - in head/sys: amd64/conf conf net netinet netinet6 sys
Message-ID:  <CAJ-Vmo=5YdLTkD0Dv5qHQokTQxtjFB=t2NqXfFzuZRLqjOOu%2Bg@mail.gmail.com>
In-Reply-To: <201403150057.s2F0vofg081606@svn.freebsd.org>
References:  <201403150057.s2F0vofg081606@svn.freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Woo! Thanks!

Adrian
 On Mar 14, 2014 5:57 PM, "Robert Watson" <rwatson@freebsd.org> wrote:

> Author: rwatson
> Date: Sat Mar 15 00:57:50 2014
> New Revision: 263198
> URL: http://svnweb.freebsd.org/changeset/base/263198
>
> Log:
>   Several years after initial development, merge prototype support for
>   linking NIC Receive Side Scaling (RSS) to the network stack's
>   connection-group implementation.  This prototype (and derived patches)
>   are in use at Juniper and several other FreeBSD-using companies, so
>   despite some reservations about its maturity, merge the patch to the
>   base tree so that it can be iteratively refined in collaboration rather
>   than maintained as a set of gradually diverging patch sets.
>
>   (1) Merge a software implementation of the Toeplitz hash specified in
>       RSS implemented by David Malone.  This is used to allow suitable
>       pcbgroup placement of connections before the first packet is
>       received from the NIC.  Software hashing is generally avoided,
>       however, due to high cost of the hash on general-purpose CPUs.
>
>   (2) In in_rss.c, maintain authoritative versions of RSS state intended
>       to be pushed to each NIC, including keying material, hash
>       algorithm/ configuration, and buckets.  Provide software-facing
>       interfaces to hash 2- and 4-tuples for IPv4 and IPv6 using both
>       the RSS standardised Toeplitz and a 'naive' variation with a hash
>       efficient in software but with poor distribution properties.
>       Implement rss_m2cpuid()to be used by netisr and other load
>       balancing code to look up the CPU on which an mbuf should be
>       processed.
>
>   (3) In the Ethernet link layer, allow netisr distribution using RSS as
>       a source of policy as an alternative to source ordering; continue
>       to default to direct dispatch (i.e., don't try and requeue packets
>       for processing on the 'right' CPU if they arrive in a directly
>       dispatchable context).
>
>   (4) Allow RSS to control tuning of connection groups in order to align
>       groups with RSS buckets.  If a packet arrives on a protocol using
>       connection groups, and contains a suitable hardware-generated
>       hash, use that hash value to select the connection group for pcb
>       lookup for both IPv4 and IPv6.  If no hardware-generated Toeplitz
>       hash is available, we fall back on regular PCB lookup risking
>       contention rather than pay the cost of Toeplitz in software --
>       this is a less scalable but, at my last measurement, faster
>       approach.  As core counts go up, we may want to revise this
>       strategy despite CPU overhead.
>
>   Where device drivers suitably configure NICs, and connection groups /
>   RSS are enabled, this should avoid both lock and line contention during
>   connection lookup for TCP.  This commit does not modify any device
>   drivers to tune device RSS configuration to the global RSS
>   configuration; patches are in circulation to do this for at least
>   Chelsio T3 and Intel 1G/10G drivers.  Currently, the KPI for device
>   drivers is not particularly robust, nor aware of more advanced features
>   such as runtime reconfiguration/rebalancing.  This will hopefully prove
>   a useful starting point for refinement.
>
>   No MFC is scheduled as we will first want to nail down a more mature
>   and maintainable KPI/KBI for device drivers.
>
>   Sponsored by:   Juniper Networks (original work)
>   Sponsored by:   EMC/Isilon (patch update and merge)
>
> Added:
>   head/sys/netinet/in_rss.c   (contents, props changed)
>   head/sys/netinet/in_rss.h   (contents, props changed)
>   head/sys/netinet/toeplitz.c   (contents, props changed)
>   head/sys/netinet/toeplitz.h   (contents, props changed)
> Modified:
>   head/sys/amd64/conf/GENERIC
>   head/sys/conf/files
>   head/sys/conf/options
>   head/sys/net/if_ethersubr.c
>   head/sys/netinet/in_pcb.c
>   head/sys/netinet/in_pcbgroup.c
>   head/sys/netinet6/in6_pcb.c
>   head/sys/netinet6/in6_pcbgroup.c
>   head/sys/sys/priv.h
>
> Modified: head/sys/amd64/conf/GENERIC
>
> ==============================================================================
> --- head/sys/amd64/conf/GENERIC Sat Mar 15 00:23:35 2014        (r263197)
> +++ head/sys/amd64/conf/GENERIC Sat Mar 15 00:57:50 2014        (r263198)
> @@ -28,6 +28,8 @@ options       SCHED_ULE               # ULE scheduler
>  options        PREEMPTION              # Enable kernel thread preemption
>  options        INET                    # InterNETworking
>  options        INET6                   # IPv6 communications protocols
> +options        PCBGROUP                # Protocol control-block groups
> +options        RSS                     # Receive-side scaling support
>  options        TCP_OFFLOAD             # TCP offload
>  options        SCTP                    # Stream Control Transmission
> Protocol
>  options        FFS                     # Berkeley Fast Filesystem
>
> Modified: head/sys/conf/files
>
> ==============================================================================
> --- head/sys/conf/files Sat Mar 15 00:23:35 2014        (r263197)
> +++ head/sys/conf/files Sat Mar 15 00:57:50 2014        (r263198)
> @@ -3267,6 +3267,7 @@ netinet/in_pcb.c          optional inet | inet6
>  netinet/in_pcbgroup.c          optional inet pcbgroup | inet6 pcbgroup
>  netinet/in_proto.c             optional inet | inet6
>  netinet/in_rmx.c               optional inet
> +netinet/in_rss.c               optional inet rss | inet6 rss
>  netinet/ip_divert.c            optional inet ipdivert ipfirewall
>  netinet/ip_ecn.c               optional inet | inet6
>  netinet/ip_encap.c             optional inet | inet6
> @@ -3308,6 +3309,7 @@ netinet/tcp_syncache.c            optional inet |
>  netinet/tcp_timer.c            optional inet | inet6
>  netinet/tcp_timewait.c         optional inet | inet6
>  netinet/tcp_usrreq.c           optional inet | inet6
> +netinet/toeplitz.c             optional inet rss | inet6 rss
>  netinet/udp_usrreq.c           optional inet | inet6
>  netinet/libalias/alias.c       optional libalias inet | netgraph_nat inet
>  netinet/libalias/alias_db.c    optional libalias inet | netgraph_nat inet
>
> Modified: head/sys/conf/options
>
> ==============================================================================
> --- head/sys/conf/options       Sat Mar 15 00:23:35 2014        (r263197)
> +++ head/sys/conf/options       Sat Mar 15 00:57:50 2014        (r263198)
> @@ -427,6 +427,7 @@ PCBGROUP            opt_pcbgroup.h
>  PF_DEFAULT_TO_DROP     opt_pf.h
>  RADIX_MPATH            opt_mpath.h
>  ROUTETABLES            opt_route.h
> +RSS                    opt_rss.h
>  SLIP_IFF_OPTS          opt_slip.h
>  TCPDEBUG
>  TCP_OFFLOAD            opt_inet.h # Enable code to dispatch TCP offloading
>
> Modified: head/sys/net/if_ethersubr.c
>
> ==============================================================================
> --- head/sys/net/if_ethersubr.c Sat Mar 15 00:23:35 2014        (r263197)
> +++ head/sys/net/if_ethersubr.c Sat Mar 15 00:57:50 2014        (r263198)
> @@ -34,6 +34,7 @@
>  #include "opt_inet6.h"
>  #include "opt_netgraph.h"
>  #include "opt_mbuf_profiling.h"
> +#include "opt_rss.h"
>
>  #include <sys/param.h>
>  #include <sys/systm.h>
> @@ -70,6 +71,7 @@
>  #include <netinet/in.h>
>  #include <netinet/in_var.h>
>  #include <netinet/if_ether.h>
> +#include <netinet/in_rss.h>
>  #include <netinet/ip_carp.h>
>  #include <netinet/ip_var.h>
>  #endif
> @@ -583,7 +585,22 @@ ether_input_internal(struct ifnet *ifp,
>
>  /*
>   * Ethernet input dispatch; by default, direct dispatch here regardless of
> - * global configuration.
> + * global configuration.  However, if RSS is enabled, hook up RSS affinity
> + * so that when deferred or hybrid dispatch is enabled, we can
> redistribute
> + * load based on RSS.
> + *
> + * XXXRW: Would be nice if the ifnet passed up a flag indicating whether
> or
> + * not it had already done work distribution via multi-queue.  Then we
> could
> + * direct dispatch in the event load balancing was already complete and
> + * handle the case of interfaces with different capabilities better.
> + *
> + * XXXRW: Sort of want an M_DISTRIBUTED flag to avoid multiple
> distributions
> + * at multiple layers?
> + *
> + * XXXRW: For now, enable all this only if RSS is compiled in, although it
> + * works fine without RSS.  Need to characterise the performance overhead
> + * of the detour through the netisr code in the event the result is always
> + * direct dispatch.
>   */
>  static void
>  ether_nh_input(struct mbuf *m)
> @@ -596,8 +613,14 @@ static struct netisr_handler       ether_nh =
>         .nh_name = "ether",
>         .nh_handler = ether_nh_input,
>         .nh_proto = NETISR_ETHER,
> +#ifdef RSS
> +       .nh_policy = NETISR_POLICY_CPU,
> +       .nh_dispatch = NETISR_DISPATCH_DIRECT,
> +       .nh_m2cpuid = rss_m2cpuid,
> +#else
>         .nh_policy = NETISR_POLICY_SOURCE,
>         .nh_dispatch = NETISR_DISPATCH_DIRECT,
> +#endif
>  };
>
>  static void
>
> Modified: head/sys/netinet/in_pcb.c
>
> ==============================================================================
> --- head/sys/netinet/in_pcb.c   Sat Mar 15 00:23:35 2014        (r263197)
> +++ head/sys/netinet/in_pcb.c   Sat Mar 15 00:57:50 2014        (r263198)
> @@ -43,6 +43,7 @@ __FBSDID("$FreeBSD$");
>  #include "opt_inet.h"
>  #include "opt_inet6.h"
>  #include "opt_pcbgroup.h"
> +#include "opt_rss.h"
>
>  #include <sys/param.h>
>  #include <sys/systm.h>
> @@ -75,6 +76,7 @@ __FBSDID("$FreeBSD$");
>  #if defined(INET) || defined(INET6)
>  #include <netinet/in.h>
>  #include <netinet/in_pcb.h>
> +#include <netinet/in_rss.h>
>  #include <netinet/ip_var.h>
>  #include <netinet/tcp_var.h>
>  #include <netinet/udp.h>
> @@ -1820,7 +1822,7 @@ struct inpcb *
>  in_pcblookup(struct inpcbinfo *pcbinfo, struct in_addr faddr, u_int fport,
>      struct in_addr laddr, u_int lport, int lookupflags, struct ifnet *ifp)
>  {
> -#if defined(PCBGROUP)
> +#if defined(PCBGROUP) && !defined(RSS)
>         struct inpcbgroup *pcbgroup;
>  #endif
>
> @@ -1829,7 +1831,17 @@ in_pcblookup(struct inpcbinfo *pcbinfo,
>         KASSERT((lookupflags & (INPLOOKUP_RLOCKPCB | INPLOOKUP_WLOCKPCB))
> != 0,
>             ("%s: LOCKPCB not set", __func__));
>
> -#if defined(PCBGROUP)
> +       /*
> +        * When not using RSS, use connection groups in preference to the
> +        * reservation table when looking up 4-tuples.  When using RSS,
> just
> +        * use the reservation table, due to the cost of the Toeplitz hash
> +        * in software.
> +        *
> +        * XXXRW: This policy belongs in the pcbgroup code, as in principle
> +        * we could be doing RSS with a non-Toeplitz hash that is
> affordable
> +        * in software.
> +        */
> +#if defined(PCBGROUP) && !defined(RSS)
>         if (in_pcbgroup_enabled(pcbinfo)) {
>                 pcbgroup = in_pcbgroup_bytuple(pcbinfo, laddr, lport,
> faddr,
>                     fport);
> @@ -1856,16 +1868,27 @@ in_pcblookup_mbuf(struct inpcbinfo *pcbi
>             ("%s: LOCKPCB not set", __func__));
>
>  #ifdef PCBGROUP
> -       if (in_pcbgroup_enabled(pcbinfo)) {
> +       /*
> +        * If we can use a hardware-generated hash to look up the
> connection
> +        * group, use that connection group to find the inpcb.  Otherwise
> +        * fall back on a software hash -- or the reservation table if
> we're
> +        * using RSS.
> +        *
> +        * XXXRW: As above, that policy belongs in the pcbgroup code.
> +        */
> +       if (in_pcbgroup_enabled(pcbinfo) &&
> +           !(M_HASHTYPE_TEST(m, M_HASHTYPE_NONE))) {
>                 pcbgroup = in_pcbgroup_byhash(pcbinfo, M_HASHTYPE_GET(m),
>                     m->m_pkthdr.flowid);
>                 if (pcbgroup != NULL)
>                         return (in_pcblookup_group(pcbinfo, pcbgroup,
> faddr,
>                             fport, laddr, lport, lookupflags, ifp));
> +#ifndef RSS
>                 pcbgroup = in_pcbgroup_bytuple(pcbinfo, laddr, lport,
> faddr,
>                     fport);
>                 return (in_pcblookup_group(pcbinfo, pcbgroup, faddr, fport,
>                     laddr, lport, lookupflags, ifp));
> +#endif
>         }
>  #endif
>         return (in_pcblookup_hash(pcbinfo, faddr, fport, laddr, lport,
>
> Modified: head/sys/netinet/in_pcbgroup.c
>
> ==============================================================================
> --- head/sys/netinet/in_pcbgroup.c      Sat Mar 15 00:23:35 2014
>  (r263197)
> +++ head/sys/netinet/in_pcbgroup.c      Sat Mar 15 00:57:50 2014
>  (r263198)
> @@ -32,6 +32,7 @@
>  __FBSDID("$FreeBSD$");
>
>  #include "opt_inet6.h"
> +#include "opt_rss.h"
>
>  #include <sys/param.h>
>  #include <sys/lock.h>
> @@ -43,6 +44,7 @@ __FBSDID("$FreeBSD$");
>
>  #include <netinet/in.h>
>  #include <netinet/in_pcb.h>
> +#include <netinet/in_rss.h>
>  #ifdef INET6
>  #include <netinet6/in6_pcb.h>
>  #endif /* INET6 */
> @@ -60,6 +62,13 @@ __FBSDID("$FreeBSD$");
>   * minimal cache line migration and lock contention during steady state
>   * operation.
>   *
> + * Hardware-offloaded checksums are often inefficient in software -- for
> + * example, Toeplitz, specified by RSS, introduced a significant overhead
> if
> + * performed during per-packge processing.  It is therefore desirable to
> fall
> + * back on traditional reservation table lookups without affinity where
> + * hardware-offloaded checksums aren't available, such as for traffic over
> + * non-RSS interfaces.
> + *
>   * Internet protocols, such as UDP and TCP, register to use connection
> groups
>   * by providing an ipi_hashfields value other than IPI_HASHFIELDS_NONE;
> this
>   * indicates to the connection group code whether a 2-tuple or 4-tuple is
> @@ -72,6 +81,11 @@ __FBSDID("$FreeBSD$");
>   * signficantly more expensive than without connection groups, but that
>   * steady-state processing can be significantly faster.
>   *
> + * When RSS is used, certain connection group parameters, such as the
> number
> + * of groups, are provided by the RSS implementation, found in in_rss.c.
> + * Otherwise, in_pcbgroup.c selects possible sensible parameters
> + * corresponding to the degree of parallelism exposed by netisr.
> + *
>   * Most of the implementation of connection groups is in this file;
> however,
>   * connection group lookup is implemented in in_pcb.c alongside
> reservation
>   * table lookups -- see in_pcblookup_group().
> @@ -126,11 +140,25 @@ in_pcbgroup_init(struct inpcbinfo *pcbin
>         if (mp_ncpus == 1)
>                 return;
>
> +#ifdef RSS
>         /*
> -        * Use one group per CPU for now.  If we decide to do dynamic
> -        * rebalancing a la RSS, we'll need to shift left by at least 1.
> +        * If we're using RSS, then RSS determines the number of connection
> +        * groups to use: one connection group per RSS bucket.  If for some
> +        * reason RSS isn't able to provide a number of buckets, disable
> +        * connection groups entirely.
> +        *
> +        * XXXRW: Can this ever happen?
> +        */
> +       numpcbgroups = rss_getnumbuckets();
> +       if (numpcbgroups == 0)
> +               return;
> +#else
> +       /*
> +        * Otherwise, we'll just use one per CPU for now.  If we decide to
> +        * do dynamic rebalancing a la RSS, we'll need similar logic here.
>          */
>         numpcbgroups = mp_ncpus;
> +#endif
>
>         pcbinfo->ipi_hashfields = hashfields;
>         pcbinfo->ipi_pcbgroups = malloc(numpcbgroups *
> @@ -146,10 +174,19 @@ in_pcbgroup_init(struct inpcbinfo *pcbin
>
>                 /*
>                  * Initialise notional affinity of the pcbgroup -- for RSS,
> -                * we want the same notion of affinity as NICs to be used.
> -                * Just round robin for the time being.
> +                * we want the same notion of affinity as NICs to be used.
>  In
> +                * the non-RSS case, just round robin for the time being.
> +                *
> +                * XXXRW: The notion of a bucket to CPU mapping is common
> at
> +                * both pcbgroup and RSS layers -- does that mean that we
> +                * should migrate it all from RSS to here, and just leave
> RSS
> +                * responsible only for providing hashing and mapping
> funtions?
>                  */
> +#ifdef RSS
> +               pcbgroup->ipg_cpu = rss_getcpu(pgn);
> +#else
>                 pcbgroup->ipg_cpu = (pgn % mp_ncpus);
> +#endif
>         }
>  }
>
> @@ -179,23 +216,38 @@ in_pcbgroup_destroy(struct inpcbinfo *pc
>
>  /*
>   * Given a hash of whatever the covered tuple might be, return a pcbgroup
> - * index.
> + * index.  Where RSS is supported, try to align bucket selection with RSS
> CPU
> + * affinity strategy.
>   */
>  static __inline u_int
>  in_pcbgroup_getbucket(struct inpcbinfo *pcbinfo, uint32_t hash)
>  {
>
> +#ifdef RSS
> +       return (rss_getbucket(hash));
> +#else
>         return (hash % pcbinfo->ipi_npcbgroups);
> +#endif
>  }
>
>  /*
>   * Map a (hashtype, hash) tuple into a connection group, or NULL if the
> hash
> - * information is insufficient to identify the pcbgroup.
> + * information is insufficient to identify the pcbgroup.  This might
> occur if
> + * a TCP packet turns up with a 2-tuple hash, or if an RSS hash is
> present but
> + * RSS is not compiled into the kernel.
>   */
>  struct inpcbgroup *
>  in_pcbgroup_byhash(struct inpcbinfo *pcbinfo, u_int hashtype, uint32_t
> hash)
>  {
>
> +#ifdef RSS
> +       if ((pcbinfo->ipi_hashfields == IPI_HASHFIELDS_4TUPLE &&
> +           hashtype == M_HASHTYPE_RSS_TCP_IPV4) ||
> +           (pcbinfo->ipi_hashfields == IPI_HASHFIELDS_2TUPLE &&
> +           hashtype == M_HASHTYPE_RSS_IPV4))
> +               return (&pcbinfo->ipi_pcbgroups[
> +                   in_pcbgroup_getbucket(pcbinfo, hash)]);
> +#endif
>         return (NULL);
>  }
>
> @@ -213,13 +265,26 @@ in_pcbgroup_bytuple(struct inpcbinfo *pc
>  {
>         uint32_t hash;
>
> +       /*
> +        * RSS note: we pass foreign addr/port as source, and local
> addr/port
> +        * as destination, as we want to align with what the hardware is
> +        * doing.
> +        */
>         switch (pcbinfo->ipi_hashfields) {
>         case IPI_HASHFIELDS_4TUPLE:
> +#ifdef RSS
> +               hash = rss_hash_ip4_4tuple(faddr, fport, laddr, lport);
> +#else
>                 hash = faddr.s_addr ^ fport;
> +#endif
>                 break;
>
>         case IPI_HASHFIELDS_2TUPLE:
> +#ifdef RSS
> +               hash = rss_hash_ip4_2tuple(faddr, laddr);
> +#else
>                 hash = faddr.s_addr ^ laddr.s_addr;
> +#endif
>                 break;
>
>         default:
>
> Added: head/sys/netinet/in_rss.c
>
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/sys/netinet/in_rss.c   Sat Mar 15 00:57:50 2014        (r263198)
> @@ -0,0 +1,505 @@
> +/*-
> + * Copyright (c) 2010-2011 Juniper Networks, Inc.
> + * All rights reserved.
> + *
> + * This software was developed by Robert N. M. Watson under contract
> + * to Juniper Networks, Inc.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> PURPOSE
> + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
> WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <sys/cdefs.h>
> +
> +__FBSDID("$FreeBSD$");
> +
> +#include "opt_inet6.h"
> +#include "opt_pcbgroup.h"
> +
> +#ifndef PCBGROUP
> +#error "options RSS depends on options PCBGROUP"
> +#endif
> +
> +#include <sys/param.h>
> +#include <sys/mbuf.h>
> +#include <sys/socket.h>
> +#include <sys/priv.h>
> +#include <sys/kernel.h>
> +#include <sys/smp.h>
> +#include <sys/sysctl.h>
> +
> +#include <net/if.h>
> +#include <net/if_var.h>
> +#include <net/netisr.h>
> +
> +#include <netinet/in.h>
> +#include <netinet/in_pcb.h>
> +#include <netinet/in_rss.h>
> +#include <netinet/in_var.h>
> +#include <netinet/toeplitz.h>
> +
> +/*-
> + * Operating system parts of receiver-side scaling (RSS), which allows
> + * network cards to direct flows to particular receive queues based on
> hashes
> + * of header tuples.  This implementation aligns RSS buckets with
> connection
> + * groups at the TCP/IP layer, so each bucket is associated with exactly
> one
> + * group.  As a result, the group lookup structures (and lock) should
> have an
> + * effective affinity with exactly one CPU.
> + *
> + * Network device drivers needing to configure RSS will query this
> framework
> + * for parameters, such as the current RSS key, hashing policies, number
> of
> + * bits, and indirection table mapping hashes to buckets and CPUs.  They
> may
> + * provide their own supplementary information, such as queue<->CPU
> bindings.
> + * It is the responsibility of the network device driver to inject packets
> + * into the stack on as close to the right CPU as possible, if playing by
> RSS
> + * rules.
> + *
> + * TODO:
> + *
> + * - Synchronization for rss_key and other future-configurable parameters.
> + * - Event handler drivers can register to pick up RSS configuration
> changes.
> + * - Should we allow rss_basecpu to be configured?
> + * - Randomize key on boot.
> + * - IPv6 support.
> + * - Statistics on how often there's a misalignment between hardware
> + *   placement and pcbgroup expectations.
> + */
> +
> +SYSCTL_NODE(_net_inet, OID_AUTO, rss, CTLFLAG_RW, 0, "Receive-side
> steering");
> +
> +/*
> + * Toeplitz is the only required hash function in the RSS spec, so use it
> by
> + * default.
> + */
> +static u_int   rss_hashalgo = RSS_HASH_TOEPLITZ;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, hashalgo, CTLFLAG_RD, &rss_hashalgo,
> 0,
> +    "RSS hash algorithm");
> +TUNABLE_INT("net.inet.rss.hashalgo", &rss_hashalgo);
> +
> +/*
> + * Size of the indirection table; at most 128 entries per the RSS spec.
>  We
> + * size it to at least 2 times the number of CPUs by default to allow
> useful
> + * rebalancing.  If not set explicitly with a loader tunable, we tune
> based
> + * on the number of CPUs present.
> + *
> + * XXXRW: buckets might be better to use for the tunable than bits.
> + */
> +static u_int   rss_bits;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, bits, CTLFLAG_RD, &rss_bits, 0,
> +    "RSS bits");
> +TUNABLE_INT("net.inet.rss.bits", &rss_bits);
> +
> +static u_int   rss_mask;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, mask, CTLFLAG_RD, &rss_mask, 0,
> +    "RSS mask");
> +
> +static const u_int     rss_maxbits = RSS_MAXBITS;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, maxbits, CTLFLAG_RD,
> +    __DECONST(int *, &rss_maxbits), 0, "RSS maximum bits");
> +
> +/*
> + * RSS's own count of the number of CPUs it could be using for processing.
> + * Bounded to 64 by RSS constants.
> + */
> +static u_int   rss_ncpus;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, ncpus, CTLFLAG_RD, &rss_ncpus, 0,
> +    "Number of CPUs available to RSS");
> +
> +#define        RSS_MAXCPUS     (1 << (RSS_MAXBITS - 1))
> +static const u_int     rss_maxcpus = RSS_MAXCPUS;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, maxcpus, CTLFLAG_RD,
> +    __DECONST(int *, &rss_maxcpus), 0, "RSS maximum CPUs that can be
> used");
> +
> +/*
> + * Variable exists just for reporting rss_bits in a user-friendly way.
> + */
> +static u_int   rss_buckets;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, buckets, CTLFLAG_RD, &rss_buckets, 0,
> +    "RSS buckets");
> +
> +/*
> + * Base CPU number; devices will add this to all CPU numbers returned by
> the
> + * RSS indirection table.  Currently unmodifable in FreeBSD.
> + */
> +static const u_int     rss_basecpu;
> +SYSCTL_INT(_net_inet_rss, OID_AUTO, basecpu, CTLFLAG_RD,
> +    __DECONST(int *, &rss_basecpu), 0, "RSS base CPU");
> +
> +/*
> + * RSS secret key, intended to prevent attacks on load-balancing.  Its
> + * effectiveness may be limited by algorithm choice and available entropy
> + * during the boot.
> + *
> + * XXXRW: And that we don't randomize it yet!
> + *
> + * XXXRW: This default is actually the default key from Chelsio T3 cards,
> as
> + * it offers reasonable distribution, unlike all-0 keys which always
> + * generate a hash of 0 (upsettingly).
> + */
> +static uint8_t rss_key[RSS_KEYSIZE] = {
> +       0x43, 0xa3, 0x8f, 0xb0, 0x41, 0x67, 0x25, 0x3d,
> +       0x25, 0x5b, 0x0e, 0xc2, 0x6d, 0x5a, 0x56, 0xda,
> +       0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> +       0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> +       0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
> +};
> +
> +/*
> + * RSS hash->CPU table, which maps hashed packet headers to particular
> CPUs.
> + * Drivers may supplement this table with a seperate CPU<->queue table
> when
> + * programming devices.
> + */
> +struct rss_table_entry {
> +       uint8_t         rte_cpu;        /* CPU affinity of bucket. */
> +};
> +static struct rss_table_entry  rss_table[RSS_TABLE_MAXLEN];
> +
> +static void
> +rss_init(__unused void *arg)
> +{
> +       u_int i;
> +
> +       /*
> +        * Validate tunables, coerce to sensible values.
> +        */
> +       switch (rss_hashalgo) {
> +       case RSS_HASH_TOEPLITZ:
> +       case RSS_HASH_NAIVE:
> +               break;
> +
> +       default:
> +               printf("%s: invalid RSS hashalgo %u, coercing to %u",
> +                   __func__, rss_hashalgo, RSS_HASH_TOEPLITZ);
> +               rss_hashalgo = RSS_HASH_TOEPLITZ;
> +       }
> +
> +       /*
> +        * Count available CPUs.
> +        *
> +        * XXXRW: Note incorrect assumptions regarding contiguity of this
> set
> +        * elsewhere.
> +        */
> +       rss_ncpus = 0;
> +       for (i = 0; i <= mp_maxid; i++) {
> +               if (CPU_ABSENT(i))
> +                       continue;
> +               rss_ncpus++;
> +       }
> +       if (rss_ncpus > RSS_MAXCPUS)
> +               rss_ncpus = RSS_MAXCPUS;
> +
> +       /*
> +        * Tune RSS table entries to be no less than 2x the number of CPUs
> +        * -- unless we're running uniprocessor, in which case there's not
> +        * much point in having buckets to rearrange for load-balancing!
> +        */
> +       if (rss_ncpus > 1) {
> +               if (rss_bits == 0)
> +                       rss_bits = fls(rss_ncpus - 1) + 1;
> +
> +               /*
> +                * Microsoft limits RSS table entries to 128, so apply that
> +                * limit to both auto-detected CPU counts and
> user-configured
> +                * ones.
> +                */
> +               if (rss_bits == 0 || rss_bits > RSS_MAXBITS) {
> +                       printf("%s: RSS bits %u not valid, coercing to
>  %u",
> +                           __func__, rss_bits, RSS_MAXBITS);
> +                       rss_bits = RSS_MAXBITS;
> +               }
> +
> +               /*
> +                * Figure out how many buckets to use; warn if less than
> the
> +                * number of configured CPUs, although this is not a fatal
> +                * problem.
> +                */
> +               rss_buckets = (1 << rss_bits);
> +               if (rss_buckets < rss_ncpus)
> +                       printf("%s: WARNING: rss_buckets (%u) less than "
> +                           "rss_ncpus (%u)\n", __func__, rss_buckets,
> +                           rss_ncpus);
> +               rss_mask = rss_buckets - 1;
> +       } else {
> +               rss_bits = 0;
> +               rss_buckets = 1;
> +               rss_mask = 0;
> +       }
> +
> +       /*
> +        * Set up initial CPU assignments: round-robin by default.
> +        *
> +        * XXXRW: Need a mapping to non-contiguous IDs here.
> +        */
> +       for (i = 0; i < rss_buckets; i++)
> +               rss_table[i].rte_cpu = i % rss_ncpus;
> +
> +       /*
> +        * Randomize rrs_key.
> +        *
> +        * XXXRW: Not yet.  If nothing else, will require an rss_isbadkey()
> +        * loop to check for "bad" RSS keys.
> +        */
> +}
> +SYSINIT(rss_init, SI_SUB_SOFTINTR, SI_ORDER_SECOND, rss_init, NULL);
> +
> +static uint32_t
> +rss_naive_hash(u_int keylen, const uint8_t *key, u_int datalen,
> +    const uint8_t *data)
> +{
> +       uint32_t v;
> +       u_int i;
> +
> +       v = 0;
> +       for (i = 0; i < keylen; i++)
> +               v += key[i];
> +       for (i = 0; i < datalen; i++)
> +               v += data[i];
> +       return (v);
> +}
> +
> +static uint32_t
> +rss_hash(u_int datalen, const uint8_t *data)
> +{
> +
> +       switch (rss_hashalgo) {
> +       case RSS_HASH_TOEPLITZ:
> +               return (toeplitz_hash(sizeof(rss_key), rss_key, datalen,
> +                   data));
> +
> +       case RSS_HASH_NAIVE:
> +               return (rss_naive_hash(sizeof(rss_key), rss_key, datalen,
> +                   data));
> +
> +       default:
> +               panic("%s: unsupported/unknown hashalgo %d", __func__,
> +                   rss_hashalgo);
> +       }
> +}
> +
> +/*
> + * Hash an IPv4 2-tuple.
> + */
> +uint32_t
> +rss_hash_ip4_2tuple(struct in_addr src, struct in_addr dst)
> +{
> +       uint8_t data[sizeof(src) + sizeof(dst)];
> +       u_int datalen;
> +
> +       datalen = 0;
> +       bcopy(&src, &data[datalen], sizeof(src));
> +       datalen += sizeof(src);
> +       bcopy(&dst, &data[datalen], sizeof(dst));
> +       datalen += sizeof(dst);
> +       return (rss_hash(datalen, data));
> +}
> +
> +/*
> + * Hash an IPv4 4-tuple.
> + */
> +uint32_t
> +rss_hash_ip4_4tuple(struct in_addr src, u_short srcport, struct in_addr
> dst,
> +    u_short dstport)
> +{
> +       uint8_t data[sizeof(src) + sizeof(dst) + sizeof(srcport) +
> +           sizeof(dstport)];
> +       u_int datalen;
> +
> +       datalen = 0;
> +       bcopy(&src, &data[datalen], sizeof(src));
> +       datalen += sizeof(src);
> +       bcopy(&dst, &data[datalen], sizeof(dst));
> +       datalen += sizeof(dst);
> +       bcopy(&srcport, &data[datalen], sizeof(srcport));
> +       datalen += sizeof(srcport);
> +       bcopy(&dstport, &data[datalen], sizeof(dstport));
> +       datalen += sizeof(dstport);
> +       return (rss_hash(datalen, data));
> +}
> +
> +#ifdef INET6
> +/*
> + * Hash an IPv6 2-tuple.
> + */
> +uint32_t
> +rss_hash_ip6_2tuple(struct in6_addr src, struct in6_addr dst)
> +{
> +       uint8_t data[sizeof(src) + sizeof(dst)];
> +       u_int datalen;
> +
> +       datalen = 0;
> +       bcopy(&src, &data[datalen], sizeof(src));
> +       datalen += sizeof(src);
> +       bcopy(&dst, &data[datalen], sizeof(dst));
> +       datalen += sizeof(dst);
> +       return (rss_hash(datalen, data));
> +}
> +
> +/*
> + * Hash an IPv6 4-tuple.
> + */
> +uint32_t
> +rss_hash_ip6_4tuple(struct in6_addr src, u_short srcport,
> +    struct in6_addr dst, u_short dstport)
> +{
> +       uint8_t data[sizeof(src) + sizeof(dst) + sizeof(srcport) +
> +           sizeof(dstport)];
> +       u_int datalen;
> +
> +       datalen = 0;
> +       bcopy(&src, &data[datalen], sizeof(src));
> +       datalen += sizeof(src);
> +       bcopy(&dst, &data[datalen], sizeof(dst));
> +       datalen += sizeof(dst);
> +       bcopy(&srcport, &data[datalen], sizeof(srcport));
> +       datalen += sizeof(srcport);
> +       bcopy(&dstport, &data[datalen], sizeof(dstport));
> +       datalen += sizeof(dstport);
> +       return (rss_hash(datalen, data));
> +}
> +#endif /* INET6 */
> +
> +/*
> + * Query the number of RSS bits in use.
> + */
> +u_int
> +rss_getbits(void)
> +{
> +
> +       return (rss_bits);
> +}
> +
> +/*
> + * Query the RSS bucket associated with an RSS hash.
> + */
> +u_int
> +rss_getbucket(u_int hash)
> +{
> +
> +       return (hash & rss_mask);
> +}
> +
> +/*
> + * Query the RSS CPU associated with an RSS bucket.
> + */
> +u_int
> +rss_getcpu(u_int bucket)
> +{
> +
> +       return (rss_table[bucket].rte_cpu);
> +}
> +
> +/*
> + * netisr CPU affinity lookup routine for use by protocols.
> + */
> +struct mbuf *
> +rss_m2cpuid(struct mbuf *m, uintptr_t source, u_int *cpuid)
> +{
> +
> +       M_ASSERTPKTHDR(m);
> +
> +       switch (M_HASHTYPE_GET(m)) {
> +       case M_HASHTYPE_RSS_IPV4:
> +       case M_HASHTYPE_RSS_TCP_IPV4:
> +               *cpuid = rss_getcpu(rss_getbucket(m->m_pkthdr.flowid));
> +               return (m);
> +
> +       default:
> +               *cpuid = NETISR_CPUID_NONE;
> +               return (m);
> +       }
> +}
> +
> +/*
> + * Query the RSS hash algorithm.
> + */
> +u_int
> +rss_gethashalgo(void)
> +{
> +
> +       return (rss_hashalgo);
> +}
> +
> +/*
> + * Query the current RSS key; likely to be used by device drivers when
> + * configuring hardware RSS.  Caller must pass an array of size
> RSS_KEYSIZE.
> + *
> + * XXXRW: Perhaps we should do the accept-a-length-and-truncate thing?
> + */
> +void
> +rss_getkey(uint8_t *key)
> +{
> +
> +       bcopy(rss_key, key, sizeof(rss_key));
> +}
> +
> +/*
> + * Query the number of buckets; this may be used by both network device
> + * drivers, which will need to populate hardware shadows of the software
> + * indirection table, and the network stack itself (such as when deciding
> how
> + * many connection groups to allocate).
> + */
> +u_int
> +rss_getnumbuckets(void)
> +{
> +
> +       return (rss_buckets);
> +}
> +
> +/*
> + * Query the number of CPUs in use by RSS; may be useful to device drivers
> + * trying to figure out how to map a larger number of CPUs into a smaller
> + * number of receive queues.
> + */
> +u_int
> +rss_getnumcpus(void)
> +{
> +
> +       return (rss_ncpus);
> +}
> +
> +/*
> + * XXXRW: Confirm that sysctl -a won't dump this keying material, don't
> want
> + * it appearing in debugging output unnecessarily.
> + */
> +static int
> +sysctl_rss_key(SYSCTL_HANDLER_ARGS)
> +{
> +       uint8_t temp_rss_key[RSS_KEYSIZE];
> +       int error;
> +
> +       error = priv_check(req->td, PRIV_NETINET_HASHKEY);
> +       if (error)
> +               return (error);
> +
> +       bcopy(rss_key, temp_rss_key, sizeof(temp_rss_key));
> +       error = sysctl_handle_opaque(oidp, temp_rss_key,
> +           sizeof(temp_rss_key), req);
> +       if (error)
> +               return (error);
> +       if (req->newptr != NULL) {
> +               /* XXXRW: Not yet. */
> +               return (EINVAL);
> +       }
> +       return (0);
> +}
> +SYSCTL_PROC(_net_inet_rss, OID_AUTO, key,
> +    CTLTYPE_OPAQUE | CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, 0, sysctl_rss_key,
> +    "", "RSS keying material");
>
> Added: head/sys/netinet/in_rss.h
>
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/sys/netinet/in_rss.h   Sat Mar 15 00:57:50 2014        (r263198)
> @@ -0,0 +1,94 @@
> +/*-
> + * Copyright (c) 2010-2011 Juniper Networks, Inc.
> + * All rights reserved.
> + *
> + * This software was developed by Robert N. M. Watson under contract
> + * to Juniper Networks, Inc.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> PURPOSE
> + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
> WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + *
> + * $FreeBSD$
> + */
> +
> +#ifndef _NETINET_IN_RSS_H_
> +#define        _NETINET_IN_RSS_H_
> +
> +#include <netinet/in.h>                /* in_addr_t */
> +
> +/*
> + * Supported RSS hash functions.
> + */
> +#define        RSS_HASH_NAIVE          0x00000001      /* Poor but fast
> hash. */
> +#define        RSS_HASH_TOEPLITZ       0x00000002      /* Required by
> RSS. */
> +#define        RSS_HASH_CRC32          0x00000004      /* Future; some
> NICs do it. */
> +
> +#define        RSS_HASH_MASK           (RSS_HASH_NAIVE |
> RSS_HASH_TOEPLITZ)
> +
> +/*
> + * Instances of struct inpcbinfo declare an RSS hash type indicating what
> + * header fields are covered.
> + */
> +#define        RSS_HASHFIELDS_NONE             0
> +#define        RSS_HASHFIELDS_4TUPLE           1
> +#define        RSS_HASHFIELDS_2TUPLE           2
> +
> +/*
> + * Compile-time limits on the size of the indirection table.
> + */
> +#define        RSS_MAXBITS     7
> +#define        RSS_TABLE_MAXLEN        (1 << RSS_MAXBITS)
> +
> +/*
> + * Maximum key size used throughout.  It's OK for hardware to use only the
> + * first 16 bytes, which is all that's required for IPv4.
> + */
> +#define        RSS_KEYSIZE     40
> +
> +/*
> + * Device driver interfaces to query RSS properties that must be
> programmed
> + * into hardware.
> + */
> +u_int  rss_getbits(void);
> +u_int  rss_getbucket(u_int hash);
> +u_int  rss_getcpu(u_int bucket);
> +void   rss_getkey(uint8_t *key);
> +u_int  rss_gethashalgo(void);
> +u_int  rss_getnumbuckets(void);
> +u_int  rss_getnumcpus(void);
> +
> +/*
> + * Network stack interface to generate a hash for a protocol tuple.
> + */
> +uint32_t       rss_hash_ip4_4tuple(struct in_addr src, u_short srcport,
> +                   struct in_addr dst, u_short dstport);
> +uint32_t       rss_hash_ip4_2tuple(struct in_addr src, struct in_addr
> dst);
> +uint32_t       rss_hash_ip6_4tuple(struct in6_addr src, u_short srcport,
> +                   struct in6_addr dst, u_short dstport);
> +uint32_t       rss_hash_ip6_2tuple(struct in6_addr src,
> +                   struct in6_addr dst);
> +
> +/*
> + * Network stack interface to query desired CPU affinity of a packet.
> + */
> +struct mbuf    *rss_m2cpuid(struct mbuf *m, uintptr_t source, u_int
> *cpuid);
> +
> +#endif /* !_NETINET_IN_RSS_H_ */
>
> Added: head/sys/netinet/toeplitz.c
>
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
> +++ head/sys/netinet/toeplitz.c Sat Mar 15 00:57:50 2014        (r263198)
> @@ -0,0 +1,58 @@
> +/*-
> + * Copyright (c) 2010 David Malone <dwmalone@FreeBSD.org>
> + * All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> PURPOSE
> + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
> STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY
> WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <sys/cdefs.h>
> +__FBSDID("$FreeBSD$");
> +
> +#include <sys/types.h>
> +
> +#include <netinet/in_rss.h>
> +#include <netinet/toeplitz.h>
> +
> +#include <sys/systm.h>
> +
> +uint32_t
> +toeplitz_hash(u_int keylen, const uint8_t *key, u_int datalen,
> +    const uint8_t *data)
> +{
> +       uint32_t hash = 0, v;
> +       u_int i, b;
> +
> +       /* XXXRW: Perhaps an assertion about key length vs. data length? */
> +
> +       v = (key[0]<<24) + (key[1]<<16) + (key[2] <<8) + key[3];
> +       for (i = 0; i < datalen; i++) {
> +               for (b = 0; b < 8; b++) {
> +                       if (data[i] & (1<<(7-b)))
> +                               hash ^= v;
> +                       v <<= 1;
> +                       if ((i + 4) < RSS_KEYSIZE &&
> +                           (key[i+4] & (1<<(7-b))))
> +                               v |= 1;
> +               }
> +       }
> +       return (hash);
> +}
>
> Added: head/sys/netinet/toeplitz.h
>
> ==============================================================================
> --- /dev/null   00:00:00 1970   (empty, because file is newly added)
>
> *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
>
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=5YdLTkD0Dv5qHQokTQxtjFB=t2NqXfFzuZRLqjOOu%2Bg>