From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 11:55:33 2013 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A9A5E83B for ; Sun, 22 Sep 2013 11:55:33 +0000 (UTC) (envelope-from mailinglists@martinlaabs.de) Received: from relay02.alfahosting-server.de (relay02.alfahosting-server.de [109.237.142.238]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6816A2AC8 for ; Sun, 22 Sep 2013 11:55:33 +0000 (UTC) Received: by relay02.alfahosting-server.de (Postfix, from userid 1001) id 6A8B332C0911; Sun, 22 Sep 2013 13:40:35 +0200 (CEST) X-Spam-DCC: : X-Spam-Level: X-Spam-Status: No, score=0.0 required=7.0 tests=BAYES_50 autolearn=disabled version=3.2.5 Received: from alfa3018.alfahosting-server.de (alfa3018.alfahosting-server.de [109.237.140.30]) by relay02.alfahosting-server.de (Postfix) with ESMTPS id 2B77632C08E7 for ; Sun, 22 Sep 2013 13:40:34 +0200 (CEST) Received: from desktop-01.martinlaabs.de (p54B33B22.dip0.t-ipconnect.de [84.179.59.34]) by alfa3018.alfahosting-server.de (Postfix) with ESMTPSA id 82DCD515C0B1 for ; Sun, 22 Sep 2013 13:40:33 +0200 (CEST) Message-ID: <523ED730.2030900@martinlaabs.de> Date: Sun, 22 Sep 2013 13:40:32 +0200 From: Martin Laabs User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130809 Thunderbird/17.0.8 MIME-Version: 1.0 To: freebsd-net@FreeBSD.org Subject: IPv6 privacy extensions breaks kerberos Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Status: No X-Virus-Checker-Version: clamassassin 1.2.4 with ClamAV 0.97.3/17883/Sun Sep 22 06:41:18 2013 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 11:55:33 -0000 Hi, I noticed that kerberos stops working when enabling the privacy extension. This is caused by the changing outgoing IP that does not fit to the dns name anymore (or do not have a dns record at all) So every host enabling the privacy extension will be unable to use kerberos and kerberos enabled services like nfs. This is a very problematic behavior and I would like to know if there is a way getting around this. Thank you, Martin Laabs From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 19:01:34 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B2C02FE8 for ; Sun, 22 Sep 2013 19:01:34 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ea0-x232.google.com (mail-ea0-x232.google.com [IPv6:2a00:1450:4013:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4BC1B20AE for ; Sun, 22 Sep 2013 19:01:34 +0000 (UTC) Received: by mail-ea0-f178.google.com with SMTP id a15so1289787eae.37 for ; Sun, 22 Sep 2013 12:01:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=OFmgiwSMZ4sRtKT/go3ei0D8GM76zDKQaEgbNmLXpUQ=; b=Llo9yx2R259qxq1IQD3mWF4ym0fy9b12U9nvRpVniYb2BETH7dKRNA37MuR58wlYpm L2HOaTnLbpaE+EVHXX8UUF02sPYM3pQvT7yBHxmJWLJ3j1UALYtCpBbObhXrVVdGeJrW 8WPJB7TcAstUAc84A/3lulH0sZc9m8aK7uF7psNvTwafGQIDseqf87VSISQ7IHzzJODH p222mIo6JyS6IFWvemXM9uNOs8TAt7b6eOYhN1oMD7V5wYk303kLrK2rNwD05UPMIAfm aKqYOSqCHT/Wgh7Cdfn9GWZFr82ErOTXoh4x2ucdYQ8VBYRrn+njIn9Mf0602xJNyWM5 jiNQ== MIME-Version: 1.0 X-Received: by 10.14.172.133 with SMTP id t5mr30020762eel.35.1379876492557; Sun, 22 Sep 2013 12:01:32 -0700 (PDT) Sender: hiren.panchasara@gmail.com Received: by 10.14.105.137 with HTTP; Sun, 22 Sep 2013 12:01:32 -0700 (PDT) Date: Sun, 22 Sep 2013 12:01:32 -0700 X-Google-Sender-Auth: UWjLaj0GLdQZuJOP-UVWZAumUFE Message-ID: Subject: Exposing sysctls for ixgbe From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 19:01:34 -0000 $ sysctl hw.igb hw.igb.rxd: 4096 hw.igb.txd: 4096 hw.igb.enable_aim: 1 hw.igb.enable_msix: 1 hw.igb.max_interrupt_rate: 8000 hw.igb.buf_ring_size: 4096 hw.igb.header_split: 0 hw.igb.num_queues: 1 hw.igb.rx_process_limit: 100 $ sysctl hw.ix sysctl: unknown oid 'hw.ix': No such file or directory I thought it would be nice to have these things exposed. So I copied them from igb: http://people.freebsd.org/~hiren/ixgbe_sysctls.txt Changes for if_igb.c is to expose correct auto-tuned value for a running system for "hw.igb.num_queues", which is not the case right now. Thanks to markj@ for help/pointers. Please let me know if the diffs look okay. cheers, Hiren From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 19:59:44 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1D1CDC67; Sun, 22 Sep 2013 19:59:44 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward-corp1g.mail.yandex.net (forward-corp1g.mail.yandex.net [IPv6:2a02:6b8:0:1402::10]) by mx1.freebsd.org (Postfix) with ESMTP id 69E7022F4; Sun, 22 Sep 2013 19:59:43 +0000 (UTC) Received: from smtpcorp4.mail.yandex.net (smtpcorp4.mail.yandex.net [95.108.252.2]) by forward-corp1g.mail.yandex.net (Yandex) with ESMTP id DD945366009A; Sun, 22 Sep 2013 23:59:40 +0400 (MSK) Received: from smtpcorp4.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp4.mail.yandex.net (Yandex) with ESMTP id 1E1F82C0058; Sun, 22 Sep 2013 23:59:40 +0400 (MSK) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp4.mail.yandex.net (nwsmtp/Yandex) with ESMTP id vYfnD1rFej-xei4WIF6; Sun, 22 Sep 2013 23:59:40 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1379879980; bh=PTdCNocghNrGKUJpHQ2f83+w9MCRgnEAdFhS+RlHncU=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject: References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=m6k8ItAvMBHUq9A03HVz2UgoLshfWFr7lzQILEivApjPY3CedK7sTAjtsEYuN6dK2 cqo8EiYKcUUqHgbvGbjhOwqPb2rZmcIQEpWaRpLv+OQdJsfpFLlXSuBZOmJWZ67rA6 lbBAKa1YMckWMMC+v7GnojZuQt6OwTrLUKzTw3+o= Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Message-ID: <523F4BED.6000804@yandex-team.ru> Date: Sun, 22 Sep 2013 23:58:37 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130824 Thunderbird/17.0.8 MIME-Version: 1.0 To: Andre Oppermann Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> <521E78B0.6080709@freebsd.org> In-Reply-To: <521E78B0.6080709@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: adrian@freebsd.org, freebsd-hackers@freebsd.org, FreeBSD Net , luigi@freebsd.org, ae@FreeBSD.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 19:59:44 -0000 On 29.08.2013 02:24, Andre Oppermann wrote: > On 28.08.2013 20:30, Alexander V. Chernikov wrote: >> Hello list! > > Hello Alexander, Hello Andre! I'm very sorry to answer so late. > > you sent quite a few things in the same email. I'll try to respond > as much as I can right now. Later you should split it up to have > more in-depth discussions on the individual parts. > > If you could make it to the EuroBSDcon 2013 DevSummit that would be > even more awesome. Most of the active network stack people will be > there too. I've sent presentation describing nearly the same things to devsummit@ so I hope this can be discussed in Networking group. I hope to attend DevSummit & EuroBSDcon. > >> There is a lot constantly raising discussions related to networking >> stack performance/changes. >> >> I'll try to summarize current problems and possible solutions from my >> point of view. >> (Generally this is one problem: stack is >> slooooooooooooooooooooooooooow, but we need to know why and >> what to do). > > Compared to others its not thaaaaaaat slow. ;) > >> Let's start with current IPv4 packet flow on a typical router: >> http://static.ipfw.ru/images/freebsd_ipv4_flow.png >> >> (I'm sorry I can't provide this as text since Visio don't have any >> 'ascii-art' exporter). >> >> Note that we are using process-to-completion model, e.g. process any >> packet in ISR until it is either >> consumed by L4+ stack or dropped or put to egress NIC queue. >> >> (There is also deferred ISR model implemented inside netisr but it >> does not change much: >> it can help to do more fine-grained hashing (for GRE or other similar >> traffic), but >> 1) it uses per-packet mutex locking which kills all performance >> 2) it currently does not have _any_ hashing functions (see absence of >> flags in `netstat -Q`) >> People using http://static.ipfw.ru/patches/netisr_ip_flowid.diff (or >> modified PPPoe/GRE version) >> report some profit, but without fixing (1) it can't help much >> ) >> >> So, let's start: >> >> 1) Ixgbe uses mutex to protect each RX ring which is perfectly fine >> since there is nearly no contention >> (the only thing that can happen is driver reconfiguration which is >> rare and, more signifficant, we >> do this once >> for the batch of packets received in given interrupt). However, due >> to some (im)possible deadlocks >> current code >> does per-packet ring unlock/lock (see ixgbe_rx_input()). >> There was a discussion ended with nothing: >> http://lists.freebsd.org/pipermail/freebsd-net/2012-October/033520.html >> >> 1*) Possible BPF users. Here we have one rlock if there are any >> readers present >> (and mutex for any matching packets, but this is more or less OK. >> Additionally, there is WIP to >> implement multiqueue BPF >> and there is chance that we can reduce lock contention there). > > Rlock to rmlock? Yes, probably. > >> There is also an "optimize_writers" hack permitting applications >> like CDP to use BPF as writers but not registering them as receivers >> (which implies rlock) > > I believe longer term we should solve this with a protocol type > "ethernet" > so that one can send/receive ethernet frames through a normal socket. Yes. AF_LINK or any similar. > >> 2/3) Virtual interfaces (laggs/vlans over lagg and other simular >> constructions). >> Currently we simply use rlock to make s/ix0/lagg0/ and, what is much >> more funny - we use complex >> vlan_hash with another rlock to >> get vlan interface from underlying one. >> >> This is definitely not like things should be done and this can be >> changed more or less easily. > > Indeed. > >> There are some useful terms/techniques in world of software/hardware >> routing: they have clear >> 'control plane' and 'data plane' separation. >> Former one is for dealing control traffic (IGP, MLD, IGMP snooping, >> lagg hellos, ARP/NDP, etc..) and >> some data traffic (packets with TTL=1, with options, destined to >> hosts without ARP/NDP record, and >> similar). Latter one is done in hardware (or effective software >> implementation). >> Control plane is responsible to provide data for efficient data plane >> operations. This is the point >> we are missing nearly everywhere. > > ACK. > >> What I want to say is: lagg is pure control-plane stuff and vlan is >> nearly the same. We can't apply >> this approach to complex cases like >> lagg-over-vlans-over-vlans-over-(pppoe_ng0-and_wifi0) >> but we definitely can do this for most common setups like (igb* or >> ix* in lagg with or without vlans >> on top of lagg). > > ACK. > >> We already have some capabilities like VLANHWFILTER/VLANHWTAG, we can >> add some more. We even have >> per-driver hooks to program HW filtering. > > We could. Though for vlan it looks like it would be easier to remove the > hardware vlan tag stripping and insertion. It only adds complexity in > all > drivers for no gain. No. Actually as far as I understand it helps driver to perform TSO. Anyway, IMO we should use HW capabilities if we can. (this probably does not add much speed on 1G, but on 10/20/40G this can help much more). > >> One small step to do is to throw packet to vlan interface directly >> (P1), proof-of-concept(working in >> production): >> http://lists.freebsd.org/pipermail/freebsd-net/2013-April/035270.html >> >> Another is to change lagg packet accounting: >> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html >> Again, this is more like HW boxes do (aggregate all counters >> including errors) (and I can't imagine >> what real error we can get from _lagg_). > > >> 4) If we are router, we can do either slooow ip_input() -> >> ip_forward() -> ip_output() cycle or use >> optimized ip_fastfwd() which falls back to 'slow' path for >> multicast/options/local traffic (e.g. >> works exactly like 'data plane' part). >> (Btw, we can consider net.inet.ip.fastforwarding to be turned on by >> default at least for non-IPSEC >> kernels) > > ACK. > >> Here we have to determine if this is local packet or not, e.g. >> F(dst_ip) returning 1 or 0. Currently >> we are simply using standard rlock + hash of iface addresses. >> (And some consumers like ipfw(4) do the same, but without lock). >> We don't need to do this! We can build sorted array of IPv4 addresses >> or other efficient structure >> on every address change and use it unlocked with delayed garbage >> collection (proof-of-concept attached) > > I'm a bit uneasy with unlocked access. On very weakly ordered > architectures > this could trip over cache coherency issues. A rmlock is essentially > for free > in the read case. Well, I'm talking of 1) allocate _new_ memory (unlocked) 2) commit _new_ copy for given address list (rlock) 3) change pointer. As fa as I understand we can read either old or new value 4) use delayed GC (how much should be wait, until deletion) Anyway, protecting (optimized) list with rmlock can do. > >> (There is another thing to discuss: maybe we can do this once >> somewhere in ip_input and mark mbuf as >> 'local/non-local' ? ) > > The problem is packet filters may change the destination address and thus > can invalidate such a lookup. Yes. So ether filter or ip_input() routing should re-inspect packet, exactly like this is done currently (ipfw fwd for IPv4/IPv6 code) > >> 5, 9) Currently we have L3 ingress/egress PFIL hooks protected by >> rmlocks. This is OK. >> >> However, 6) and 7) are not. >> Firewall can use the same pfil lock as reader protection without >> imposing its own lock. currently >> pfil&ipfw code is ready to do this. > > The problem with the global pfil rmlock is the comparatively long time it > is held in a locked state. Also packet filters may have to acquire > additional > locks when they have to modify state tables. Rmlocks are not made for > that > because they pin the thread to the cpu they're currently on. This is > what > Gleb is complaining about. Yes, additional locks is the problem > > My idea is to hold the pfil rmlock only for the lookup of the first/next > packet filter that will run, not for the entire duration. That would > solve > the problem. However packets filter then have to use their own locks > again, > which could be rmlock too. Well, we haven't changed anything yet :) > >> 8) Radix/rt* api. This is probably the worst place in entire stack. >> It is toooo generic, tooo slow >> and buggy (do you use IPv6? you definitely know what I'm talking about). >> A) It really is too generic and assumption that it can be >> (effectively) used for every family is >> wrong. Two examples: >> we don't need to lookup all 128 bits of IPv6 address. Subnets with >> mask >/64 are not used widely >> (actually the only reason to use them are p2p links due to ND >> potential problems). >> One of common solutions is to lookup 64bits, and build another trie >> (or other structure) in case of >> collision. >> Another example is MPLS where we can simply do direct array lookup >> based on ingress label. > > Yes. While we shouldn't throw it out, it should be run as RIB and > allow a much more protocol specific FIB for the hot packet path. > >> B) It is terribly slow (AFAIR luigi@ did some performance management, >> numbers available in one of >> netmap pdfs) > > Again not thaaaat slow but inefficient enough. I've found the paper I was talking about: http://info.iet.unipi.it/~luigi/papers/20120601-dxr.pdf It claims that our radix is able to do 6MPPS/core and it does not scale with number of cores. > >> C) It is not multipath-capable. Stateful (and non-working) multipath >> is definitely not the right way. > > Indeed. > >> 8*) rtentry >> We are doing it wrong. >> Currently _every_ lookup locks/unlocks given rte twice. >> First lock is related to and old-old story for trusting IP redirects >> (and auto-adding host routes >> for them). Hopefully currently it is disabled automatically when you >> turn forwarding on. > > They're disabled. > >> The second one is much more complicated: we are assuming that rte's >> with non-zero refcount value can >> stop egress interface from being destroyed. >> This is wrong (but widely used) assumption. > > Not really. The reason for the refcount is not the ifp reference but > other code parts that may hold direct pointers to the rtentry and do > direct dereferencing to access information in it. Yes, but what information? > >> We can use delayed GC instead of locking for rte's and this won't >> break things more than they are >> broken now (patch attached). > > Nope. Delayed GC is not the way to go here. To do away with rtentry > locking and refcounting we have change rtalloc(9) to return the > information > the caller wants (e.g. ifp, ia, others) and not the rtentry address > anymore. > So instead of rtalloc() we have rtlookup(). It depends on what we want to do next.. My idea (briefly) is to have 1) adjacency/nhops structures describing next hops with rewrite info and list of iface indices to do L2 multipath 2) "rtentry" to have link to array of nhops to do L3 multipath (more or less the same as Cisco CEF and others). And, anyway, we still have to protect from interface departure. > >> We can't do the same for ifp structures since >> a) virtual ones can assume some state in underlying physical NIC >> b) physical ones just _can_ be destroyed (maybe regardless of user >> wants this or not, like: SFP >> being unplugged from NIC) or simply lead to kernel crash due to SW/HW >> inconsistency > > Here I actually believe we can do a GC or stable storage based approach. > Ifp pointers are kept in too many places and properly refcounting it is > very (too) hard. So whenever an interface gets destroyed or disappears > it's callable function pointers are replaced with dummies returning an > error. The ifp in memory will stay for some time and even may be reused Yes. But we are not holding any (relevant) lock while doing actual transmit (e.g. calling if_output after performing L2 rewrite in ether_output) so some cores will see old pointers.. > for another new interface later again (Cisco does it that way in their > IOS). > >> One of possible solution is to implement stable refcounts based on >> PCPU counters, and apply thos >> counters to ifp, but seem to be non-trivial. >> >> >> Another rtalloc(9) problem is the fact that radix is used as both >> 'control plane' and 'data plane' >> structure/api. Some users always want to put more information in rte, >> while others >> want to make rte more compact. We just need _different_ structures >> for that. > > ACK. > >> Feature-rich, lot-of-data control plane one (to store everything we >> want to store, including, for >> example, PID of process originating the route) - current radix can be >> modified to do this. >> And address-family-depended another structure (array, trie, or >> anything) which contains _only_ data >> necessary to put packet on the wire. > > ACK. > >> 11) arpresolve. Currently (this was decoupled in 8.x) we have >> a) ifaddr rlock >> b) lle rlock. >> >> We don't need those locks. >> We need to >> a) make lle layer per-interface instead of global (and this can also >> solve multiple fibs and L2 >> mappings done in fib.0 issue) > > Yes! > >> b) use rtalloc(9)-provided lock instead of separate locking > > No. Interface rmlock. Discussable :) > >> c) actually, we need to do rewrite this layer because >> d) lle actually is the place to do real multipath: > > No, you can do multipath through more than one interface. If lle is > per interface that wont work and is not the right place. > >> briefly, >> you have rte pointing to some special nexthop structure pointing to >> lle, which has the following data: >> num_of_egress_ifaces: [ifindex1, ifindex2, ifindex3] | L2 data to >> prepend to header >> Separate post will follow. > > This should be part of the RIB/FIB and select on of the ifp+nexthops > to return on lookup. Yes. > >> With the following, we can achieve lagg traffic distribution without >> actually using lagg_transmit >> and similar stuff (at least in most common scenarious) > > This seems to be a rather nasty layering violation. Not really. lagg is pure virtual stuff. > >> (for example, TCP output definitely can benefit from this, since we >> can account flowid once for TCP >> session and use in in every mbuf) > > >> So. Imagine we have done all this. How we can estimate the difference? >> >> There was a thread, started a year ago, describing 'stock' >> performance and difference for various >> modifications. >> It is done on 8.x, however I've got similar results on recent 9.x >> >> http://lists.freebsd.org/pipermail/freebsd-net/2012-July/032680.html >> >> Briefly: >> >> 2xE5645 @ Intel 82599 NIC. >> Kernel: FreeBSD-8-S r237994, stock drivers, stock routing, no >> FLOWTABLE, no firewallIxia XM2 >> (traffic generator) <> ix0 (FreeBSD). Ixia sends 64byte IP packets >> from vlan10 (10.100.0.64 - >> 10.100.0.156) to destinations in vlan11 (10.100.1.128 - >> 10.100.1.192). Static arps are configured >> for all destination addresses. Traffic level is slightly above or >> slightly below system performance. >> >> we start from 1.4MPPS (if we are using several routes to minimize >> mutex contention). >> >> My 'current' result for the same test, on same HW, with the following >> modifications: >> >> * 1) ixgbe per-packet ring unlock removed >> * P1) ixgbe is modified to do direct vlan input (so 2,3 are not used) >> * 4) separate lockless in_localip() version >> * 6) - using existing pfil lock >> * 7) using lockless version >> * 8) radix converted to use rmlock instead of rlock. Delayed GC is >> used instead of mutexes >> * 10) - using existing pfil lock >> * 11) using radix lock to do arpresolve(). Not using lle rlock >> >> (so the rmlocks are the only locks used on data path). >> >> Additionally, ipstat counters are converted to PCPU (no real >> performance implications). >> ixgbe does not do per-packet accounting (as in head). >> if_vlan counters are converted to PCPU >> lagg is converted to rmlock, per-packet accounting is removed (using >> stat from underlying interfaces) >> lle hash size is bumped to 1024 instead of 32 (not applicable here, >> but slows things down for large >> L2 domains) >> >> The result is 5.6 MPPS for single port (11 cores) and 6.5MPPS for >> lagg (16 cores), nearly the same >> for HT on and 22 cores. > > That's quite good, but we want more. ;) > >> .. >> while Intel DPDK claims 80MPPS (and 6windgate talks about 160 or so) >> on the same-class hardware and >> _userland_ forwarding. > > Those numbers sound a bit far out. Maybe if the packet isn't touched > or looked at at all in a pure netmap interface to interface bridging > scenario. I don't believe these numbers. http://www.intel.com/content/dam/www/public/us/en/documents/presentation/dpdk-packet-processing-ia-overview-presentation.pdf Luigi talks about very fast L4 lookups in his (and other colleagues) work. Anyway, even simple 8-8-8-8 multi-bit trie can be very fast > >> One of key features making all such products possible (DPDK, netmap, >> packetshader, Cisco SW >> forwarding) - is use of batching instead of process-to-completion model. >> Batching mitigates locking cost, batching does not wash out CPU >> cache, and so on. > > The work has to be done eventually. Batching doesn't relieve from it. > IMHO batch moving is only the last step would should look at. It makes > the stack rather complicated and introduces other issues like packet > latency. > >> So maybe we can consider passing batches from NIC to at least L2 >> layer with netisr? or even up to >> ip_input() ? > > And then? You probably won't win much in the end (if the lock path > is optimized). At least I can firewall them "all at once". Next steps depends on how we can solve egress ifp problem. But yes, this is definitely not the first thing to do. > >> Another question is about making some sort of reliable GC like >> ("passive serialization" or other >> similar not-to-pronounce-words about Linux and lockless objects). > > Rmlocks are our secret weapon and just as good. > >> P.S. Attached patches are 1) for 8.x 2) mostly 'hacks' showing >> roughly how can this be done and what >> benefit can be achieved. > From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 20:02:22 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D762FFB8; Sun, 22 Sep 2013 20:02:22 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward-corp1g.mail.yandex.net (forward-corp1g.mail.yandex.net [IPv6:2a02:6b8:0:1402::10]) by mx1.freebsd.org (Postfix) with ESMTP id 827D92358; Sun, 22 Sep 2013 20:02:22 +0000 (UTC) Received: from smtpcorp4.mail.yandex.net (smtpcorp4.mail.yandex.net [95.108.252.2]) by forward-corp1g.mail.yandex.net (Yandex) with ESMTP id C54B9366009A; Mon, 23 Sep 2013 00:02:20 +0400 (MSK) Received: from smtpcorp4.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp4.mail.yandex.net (Yandex) with ESMTP id 2EF512C0337; Mon, 23 Sep 2013 00:02:20 +0400 (MSK) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp4.mail.yandex.net (nwsmtp/Yandex) with ESMTP id sZ8d66c12V-2KiurqgF; Mon, 23 Sep 2013 00:02:20 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1379880140; bh=ckqWsTOIe/Kq2fUVwOM8tdSMGh1wrt74DVOIcVSoROQ=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject: References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=UqhwclJ5XqEh3ojfScL1sDTZYC6IyJM1t6cWll5wv/fCnjrOTu4KyAfG/wQzY9rkf FNwVc0pgaxTowGHI03/Hgr3+l9ByEYnLs4ZAX36XWRwghuhfnKgXNNKA8RXVJZ8xyb lztFaDePItV+hzB91UQp9tJrQKpHpBB8VWUc+YRI= Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Message-ID: <523F4C8D.6080903@yandex-team.ru> Date: Mon, 23 Sep 2013 00:01:17 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130824 Thunderbird/17.0.8 MIME-Version: 1.0 To: Slawa Olhovchenkov Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> <521E78B0.6080709@freebsd.org> <20130829013241.GB70584@zxy.spb.ru> In-Reply-To: <20130829013241.GB70584@zxy.spb.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: adrian@freebsd.org, Andre Oppermann , freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, luigi@freebsd.org, ae@FreeBSD.org, FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 20:02:22 -0000 On 29.08.2013 05:32, Slawa Olhovchenkov wrote: > On Thu, Aug 29, 2013 at 12:24:48AM +0200, Andre Oppermann wrote: > >>> .. >>> while Intel DPDK claims 80MPPS (and 6windgate talks about 160 or so) on the same-class hardware and >>> _userland_ forwarding. >> Those numbers sound a bit far out. Maybe if the packet isn't touched >> or looked at at all in a pure netmap interface to interface bridging >> scenario. I don't believe these numbers. > 80*64*8 = 40.960 Gb/s > May be DCA? And use CPU with 40 PCIe lane and 4 memory chanell. Intel introduces DDIO instead of DCA: http://www.intel.com/content/www/us/en/io/direct-data-i-o.html (and it seems DCA does not help much): https://www.myricom.com/software/myri10ge/790-how-do-i-enable-intel-direct-cache-access-dca-with-the-linux-myri10ge-driver.html https://www.myricom.com/software/myri10ge/783-how-do-i-get-the-best-performance-with-my-myri-10g-network-adapters-on-a-host-that-supports-intel-data-direct-i-o-ddio.html (However, DPDK paper notes DDIO is of signifficant helpers) From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 20:13:11 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 956362E0; Sun, 22 Sep 2013 20:13:11 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward-corp1f.mail.yandex.net (forward-corp1f.mail.yandex.net [IPv6:2a02:6b8:0:801::10]) by mx1.freebsd.org (Postfix) with ESMTP id 0AEAD23F6; Sun, 22 Sep 2013 20:13:11 +0000 (UTC) Received: from smtpcorp4.mail.yandex.net (smtpcorp4.mail.yandex.net [95.108.252.2]) by forward-corp1f.mail.yandex.net (Yandex) with ESMTP id 9743C2420022; Mon, 23 Sep 2013 00:13:07 +0400 (MSK) Received: from smtpcorp4.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp4.mail.yandex.net (Yandex) with ESMTP id CB2EB2C032B; Mon, 23 Sep 2013 00:13:06 +0400 (MSK) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp4.mail.yandex.net (nwsmtp/Yandex) with ESMTP id zPyEtHgBff-D6iqY3ni; Mon, 23 Sep 2013 00:13:06 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1379880786; bh=lc1BwebjnC7LP0vJSar/INsu9fAdxzJlXjhZRROe3wc=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject: References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=TiFhbX28OlyJCId0ZK+VwWQp36YyQ+h5WJh3zgC8KN+0g0HD9q83tsYSnh8yv3F8L xX3AllLMhINn8mDOk4NZSsUIRGpnNhckim3gXJVni//gJQRXbaQZLY79Q45TEGh/pn 8RzHV1D9iadYLhu84SNJrkjRmQNZV1G2ug+v8DQo= Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Message-ID: <523F4F14.9090404@yandex-team.ru> Date: Mon, 23 Sep 2013 00:12:04 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130824 Thunderbird/17.0.8 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Luigi Rizzo , Andre Oppermann , "freebsd-hackers@freebsd.org" , FreeBSD Net , "Andrey V. Elsukov" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 20:13:11 -0000 On 29.08.2013 15:49, Adrian Chadd wrote: > Hi, Hello Adrian! I'm very sorry for the looong reply. > > There's a lot of good stuff to review here, thanks! > > Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to > keep locking things like that on a per-packet basis. We should be able > to do this in a cleaner way - we can defer RX into a CPU pinned > taskqueue and convert the interrupt handler to a fast handler that > just schedules that taskqueue. We can ignore the ithread entirely here. > > What do you think? Well, it sounds good :) But performance numbers and Jack opinion is more important :) Are you going to Malta? > > Totally pie in the sky handwaving at this point: > > * create an array of mbuf pointers for completed mbufs; > * populate the mbuf array; > * pass the array up to ether_demux(). > > For vlan handling, it may end up populating its own list of mbufs to > push up to ether_demux(). So maybe we should extend the API to have a > bitmap of packets to actually handle from the array, so we can pass up > a larger array of mbufs, note which ones are for the destination and > then the upcall can mark which frames its consumed. > > I specifically wonder how much work/benefit we may see by doing: > > * batching packets into lists so various steps can batch process > things rather than run to completion; > * batching the processing of a list of frames under a single lock > instance - eg, if the forwarding code could do the forwarding lookup > for 'n' packets under a single lock, then pass that list of frames up > to inet_pfil_hook() to do the work under one lock, etc, etc. I'm thinking the same way, but we're stuck with 'forwarding lookup' due to problem with egress interface pointer, as I mention earlier. However it is interesting to see how much it helps, regardless of locking. Currently I'm thinking that we should try to change radix to something different (it seems that it can be checked fast) and see what happened. Luigi's performance numbers for our radix are too awful, and there is a patch implementing alternative trie: http://info.iet.unipi.it/~luigi/papers/20120601-dxr.pdf http://www.nxlab.fer.hr/dxr/stable_8_20120824.diff > > Here, the processing would look less like "grab lock and process to > completion" and more like "mark and sweep" - ie, we have a list of > frames that we mark as needing processing and mark as having been > processed at each layer, so we know where to next dispatch them. > > I still have some tool coding to do with PMC before I even think about > tinkering with this as I'd like to measure stuff like per-packet > latency as well as top-level processing overhead (ie, > CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC > interrupts on that core, etc.) That will be great to see! > > Thanks, > > > > -adrian > From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 22:04:47 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5CC69A14; Sun, 22 Sep 2013 22:04:47 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 311AF299F; Sun, 22 Sep 2013 22:04:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r8MM4lTM083014; Sun, 22 Sep 2013 22:04:47 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r8MM4kZF083013; Sun, 22 Sep 2013 22:04:46 GMT (envelope-from linimon) Date: Sun, 22 Sep 2013 22:04:46 GMT Message-Id: <201309222204.r8MM4kZF083013@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/182297: [cm] ArcNet driver fails to detect the link address - and does not work at all X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 22:04:47 -0000 Old Synopsis: ArcNet driver fails to detect the link address - and does not work at all New Synopsis: [cm] ArcNet driver fails to detect the link address - and does not work at all Responsible-Changed-From-To: freebsd-i386->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Sun Sep 22 22:03:01 UTC 2013 Responsible-Changed-Why: Reclassify and assign. Note to submitter: I haven't heard of any ArcNet cards in a long time. Unfortunately, you may be the only person who is in a position to debug and fix this. http://www.freebsd.org/cgi/query-pr.cgi?pr=182297 From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 22:06:04 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id ACEF0C12; Sun, 22 Sep 2013 22:06:04 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 68C9E29BF; Sun, 22 Sep 2013 22:06:04 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1VNrp0-000Llg-HY; Mon, 23 Sep 2013 02:08:06 +0400 Date: Mon, 23 Sep 2013 02:08:06 +0400 From: Slawa Olhovchenkov To: "Alexander V. Chernikov" Subject: Re: Network stack changes Message-ID: <20130922220806.GK3796@zxy.spb.ru> References: <521E41CB.30700@yandex-team.ru> <521E78B0.6080709@freebsd.org> <20130829013241.GB70584@zxy.spb.ru> <523F4C8D.6080903@yandex-team.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <523F4C8D.6080903@yandex-team.ru> User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: adrian@freebsd.org, Andre Oppermann , freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, luigi@freebsd.org, ae@FreeBSD.org, FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 22:06:04 -0000 On Mon, Sep 23, 2013 at 12:01:17AM +0400, Alexander V. Chernikov wrote: > On 29.08.2013 05:32, Slawa Olhovchenkov wrote: > > On Thu, Aug 29, 2013 at 12:24:48AM +0200, Andre Oppermann wrote: > > > >>> .. > >>> while Intel DPDK claims 80MPPS (and 6windgate talks about 160 or so) on the same-class hardware and > >>> _userland_ forwarding. > >> Those numbers sound a bit far out. Maybe if the packet isn't touched > >> or looked at at all in a pure netmap interface to interface bridging > >> scenario. I don't believe these numbers. > > 80*64*8 = 40.960 Gb/s > > May be DCA? And use CPU with 40 PCIe lane and 4 memory chanell. > Intel introduces DDIO instead of DCA: > http://www.intel.com/content/www/us/en/io/direct-data-i-o.html > (and it seems DCA does not help much): > https://www.myricom.com/software/myri10ge/790-how-do-i-enable-intel-direct-cache-access-dca-with-the-linux-myri10ge-driver.html > https://www.myricom.com/software/myri10ge/783-how-do-i-get-the-best-performance-with-my-myri-10g-network-adapters-on-a-host-that-supports-intel-data-direct-i-o-ddio.html > > (However, DPDK paper notes DDIO is of signifficant helpers) Ha, Intel paper say SMT is signifficant better HT. In real word -- same shit. For network application, if buffring need more then L3 cache, what happening? May be some bad things... From owner-freebsd-net@FreeBSD.ORG Sun Sep 22 20:16:21 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 53BB2667; Sun, 22 Sep 2013 20:16:21 +0000 (UTC) (envelope-from melifaro@yandex-team.ru) Received: from forward-corp1e.mail.yandex.net (forward-corp1e.mail.yandex.net [IPv6:2a02:6b8:0:202::10]) by mx1.freebsd.org (Postfix) with ESMTP id EC30D242B; Sun, 22 Sep 2013 20:16:20 +0000 (UTC) Received: from smtpcorp4.mail.yandex.net (smtpcorp4.mail.yandex.net [95.108.252.2]) by forward-corp1e.mail.yandex.net (Yandex) with ESMTP id 95BAF640D0A; Mon, 23 Sep 2013 00:16:17 +0400 (MSK) Received: from smtpcorp4.mail.yandex.net (localhost [127.0.0.1]) by smtpcorp4.mail.yandex.net (Yandex) with ESMTP id 581372C032B; Mon, 23 Sep 2013 00:16:16 +0400 (MSK) Received: from dhcp170-36-red.yandex.net (dhcp170-36-red.yandex.net [95.108.170.36]) by smtpcorp4.mail.yandex.net (nwsmtp/Yandex) with ESMTP id jtjruyBg15-GGiaTnd3; Mon, 23 Sep 2013 00:16:16 +0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1379880976; bh=gvzJN62AvNEgGoGNHIYfEvCzLEcKQ9CJd4v9pZemI00=; h=Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject: References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=Gc+87YUS6ZIHr6f8OibtRuhzdovntf4M5P1jnv32kAeNRLqwz3/oJ2UxeURkPMrID rfYovH/KAOgmb5HMFh3SgBd2fcOQME6BrBubz6ELgGGcqEiBEEevu5QbGoqcve1PRb tuPOPlllk/24RD9I7z4/G5jCBW9yrIuBlc0AVoao= Authentication-Results: smtpcorp4.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Message-ID: <523F4FD1.6060807@yandex-team.ru> Date: Mon, 23 Sep 2013 00:15:13 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130824 Thunderbird/17.0.8 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Olivier_Cochard-Labb=E9?= Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> <6BDA4619-783C-433E-9819-A7EAA0BD3299@neville-neil.com> <20130914142802.GC71010@onelab2.iet.unipi.it> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sun, 22 Sep 2013 22:25:49 +0000 Cc: Adrian Chadd , Andre Oppermann , "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" , Luigi Rizzo , "Andrey V. Elsukov" , FreeBSD Net , Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Sep 2013 20:16:21 -0000 On 14.09.2013 22:49, Olivier Cochard-Labbé wrote: > On Sat, Sep 14, 2013 at 4:28 PM, Luigi Rizzo wrote: >> IXIA ? For the timescales we need to address we don't need an IXIA, >> a netmap sender is more than enough >> > The great netmap generates only one IP flow (same src/dst IP and same > src/dst port). > This don't permit to test multi-queue NIC (or SMP packet-filter) on a > simple lab like this: > netmap sender => freebsd router => netmap receiver I've got the variant which is capable on doing linerate pcap replays on single queue. (However this is true for small pcaps only) > > Regards, > > Olivier From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 03:56:03 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 632615A8 for ; Mon, 23 Sep 2013 03:56:03 +0000 (UTC) (envelope-from ihsan@grep.my) Received: from svc02-kul.b.n3labs.my (svc02-kul.b.n3labs.my [IPv6:2400:3700:10::61]) by mx1.freebsd.org (Postfix) with ESMTP id 24F4129A5 for ; Mon, 23 Sep 2013 03:56:02 +0000 (UTC) Received: from [IPv6:2400:3700:49::ddbd:7810:1737:c59b] (unknown [IPv6:2400:3700:49:0:ddbd:7810:1737:c59b]) by svc02-kul.b.n3labs.my (Postfix) with ESMTPSA id F1540C60235 for ; Mon, 23 Sep 2013 11:55:57 +0800 (MYT) From: Ihsan Junaidi Ibrahim Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Programmatically forwarding packets to outgoing interface Message-Id: <4DC4F001-D430-4110-81DA-279F3D01AD33@grep.my> Date: Mon, 23 Sep 2013 11:55:56 +0800 To: "freebsd-net@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) X-Mailer: Apple Mail (2.1510) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 03:56:03 -0000 Hi folks, I'm trying to learn building a VPN-type application on FreeBSD and I'm = currently stuck at trying to route packets to outgoing interface. I've managed to push/pop IP packets in a tun(4) interface but now that I = can read the inner packet header, I need to route the payload out of the = box. I'm not quite sure which API I need to use to achieve this. The inner packets can be of either IPv4 or IPv6. Thanks.= From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 04:10:14 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7AD9F6E1 for ; Mon, 23 Sep 2013 04:10:14 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4CF842A1E for ; Mon, 23 Sep 2013 04:10:14 +0000 (UTC) Received: from jre-mbp.elischer.org (ppp121-45-245-177.lns20.per2.internode.on.net [121.45.245.177]) (authenticated bits=0) by vps1.elischer.org (8.14.6/8.14.6) with ESMTP id r8N4AATO023018 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 22 Sep 2013 21:10:12 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <523FBF24.50004@freebsd.org> Date: Mon, 23 Sep 2013 12:10:12 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Ihsan Junaidi Ibrahim Subject: Re: Programmatically forwarding packets to outgoing interface References: <4DC4F001-D430-4110-81DA-279F3D01AD33@grep.my> In-Reply-To: <4DC4F001-D430-4110-81DA-279F3D01AD33@grep.my> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 04:10:14 -0000 On 9/23/13 11:55 AM, Ihsan Junaidi Ibrahim wrote: > Hi folks, > > I'm trying to learn building a VPN-type application on FreeBSD and I'm currently stuck at trying to route packets to outgoing interface. > > I've managed to push/pop IP packets in a tun(4) interface but now that I can read the inner packet header, I need to route the payload out of the box. I'm not quite sure which API I need to use to achieve this. > > The inner packets can be of either IPv4 or IPv6. > > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > you can try use ipfw and its 'fwd' option to reroute packets not sure if fwd works with ipv6.. I've never tried.. From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 04:42:35 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D0597AD1; Mon, 23 Sep 2013 04:42:35 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 836B62B50; Mon, 23 Sep 2013 04:42:34 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id m15so2690852wgh.9 for ; Sun, 22 Sep 2013 21:42:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=P2w1WbOvUNizXn7ZsZS2TBooYqeg4cGtYAdYuHYhYXM=; b=QOwrZB5l04e+6jyPEqzodo92zbc+lgK+Acasb9IebLx+0pMt++4YSkD1Y69XGeJN47 O5q78g4joGcHZCXb4Sn5q/+b8pgFZzoqCX3HbDHnjWLGeLaeIzvsMPRMJ2QjUd+DPgI4 kQeWEDgL2puChSMNZNc0GP+Yrc5jMlRtlX16RU2tYuX0zwZ74ydipyKtRv7zVzP5SD2i QNIdpI9rcwdFWjQBTbPx24KSQcZX+2ykEAL8KEFOJpfUIDs6zPXLWk25mKbutnQm3DZW SKWgHGVY1h0IpgLyJvTbTq+B4vDL1URAj6tEcZU0Ne9IVvTY0BRjmiX7vQU/6ObMDegy GoVA== MIME-Version: 1.0 X-Received: by 10.180.10.136 with SMTP id i8mr11840167wib.46.1379911352875; Sun, 22 Sep 2013 21:42:32 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Sun, 22 Sep 2013 21:42:32 -0700 (PDT) In-Reply-To: <523F4F14.9090404@yandex-team.ru> References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> Date: Sun, 22 Sep 2013 21:42:32 -0700 X-Google-Sender-Auth: Xk5o-T54T2C85UHlUNeXvbPizgo Message-ID: Subject: Re: Network stack changes From: Adrian Chadd To: "Alexander V. Chernikov" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Luigi Rizzo , Andre Oppermann , "freebsd-hackers@freebsd.org" , FreeBSD Net , "Andrey V. Elsukov" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 04:42:36 -0000 Hi! On 22 September 2013 13:12, Alexander V. Chernikov wrote: > I'm thinking the same way, but we're stuck with 'forwarding lookup' due > to problem with egress interface pointer, as I mention earlier. However it > is interesting to see how much it helps, regardless of locking. > > Currently I'm thinking that we should try to change radix to something > different (it seems that it can be checked fast) and see what happened. > Luigi's performance numbers for our radix are too awful, and there is a > patch implementing alternative trie: > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff > > So, I can make educated guesses about why this is better for forwarding workloads. I'd like to characterize it though. So, what's it doing that's better? better locking? better caching behaviour? less memory lookups? etc. Thanks, -adrian From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 04:42:37 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 40289AD6; Mon, 23 Sep 2013 04:42:37 +0000 (UTC) (envelope-from ihsan@grep.my) Received: from svc02-kul.b.n3labs.my (svc02-kul.b.n3labs.my [IPv6:2400:3700:10::61]) by mx1.freebsd.org (Postfix) with ESMTP id 001F42B51; Mon, 23 Sep 2013 04:42:36 +0000 (UTC) Received: from [192.168.2.176] (unknown [110.159.249.234]) by svc02-kul.b.n3labs.my (Postfix) with ESMTPSA id 492E3C60235; Mon, 23 Sep 2013 12:42:36 +0800 (MYT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Programmatically forwarding packets to outgoing interface From: Ihsan Junaidi Ibrahim In-Reply-To: <523FBF24.50004@freebsd.org> Date: Mon, 23 Sep 2013 12:42:34 +0800 Content-Transfer-Encoding: quoted-printable Message-Id: <3F7351D9-A2A9-492C-887A-E1B81083378E@grep.my> References: <4DC4F001-D430-4110-81DA-279F3D01AD33@grep.my> <523FBF24.50004@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.1510) Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 04:42:37 -0000 Thanks. Is there a specific C API I can use to call this? On Sep 23, 2013, at 12:10 PM, Julian Elischer = wrote: > On 9/23/13 11:55 AM, Ihsan Junaidi Ibrahim wrote: >> Hi folks, >>=20 >> I'm trying to learn building a VPN-type application on FreeBSD and = I'm currently stuck at trying to route packets to outgoing interface. >>=20 >> I've managed to push/pop IP packets in a tun(4) interface but now = that I can read the inner packet header, I need to route the payload out = of the box. I'm not quite sure which API I need to use to achieve this. >>=20 >> The inner packets can be of either IPv4 or IPv6. >>=20 >> Thanks. >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>=20 >>=20 > you can try use ipfw and its 'fwd' option to reroute packets > not sure if fwd works with ipv6.. I've never tried.. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 04:53:01 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 32934F5A; Mon, 23 Sep 2013 04:53:01 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A92B82BDC; Mon, 23 Sep 2013 04:52:59 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id ec20so2184616lab.28 for ; Sun, 22 Sep 2013 21:52:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=MeHddLeaytEfYArPlVU1mp/TWDEAnx+W3lRre+4umrg=; b=mLh6Xn3QSQHmiPy/AlwAozxnVJuN8aKo13h01Qfe7NhM1x7Oi1uHpgFvYsiFevA9VE N/oFFLJ1Rq84pzUe9Yk/CRHP3yHNkp3gf07JEd5PK3JDXIMERNhb6EoPlORmNocVSOFU AKa+whlt8rGV0th8mE/6+4xHVw1EJtiYTpLIZi2Hse8N+X0Ty8Wu3RMpmU+4dwZAXpdw wtsyN2jfOZheI/2/VHtya/N8o8Po/BfWidKrQiCZt+KG9W4hSrTwpXax57RgtSAPHf3+ BGIqaeXSl206878+YWdYKnOd68iysTIkf1WIGa9U9YbVaL3+7u9qsTaz0iPVUyEYCK5a o5jQ== MIME-Version: 1.0 X-Received: by 10.152.3.201 with SMTP id e9mr4930591lae.24.1379911977606; Sun, 22 Sep 2013 21:52:57 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.105 with HTTP; Sun, 22 Sep 2013 21:52:57 -0700 (PDT) In-Reply-To: References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> Date: Mon, 23 Sep 2013 06:52:57 +0200 X-Google-Sender-Auth: 2EGu8K7yJexNmLEWiHJFpHqzXNA Message-ID: Subject: Re: Network stack changes From: Luigi Rizzo To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , Luigi Rizzo , Andre Oppermann , "Andrey V. Elsukov" , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 04:53:01 -0000 On Mon, Sep 23, 2013 at 6:42 AM, Adrian Chadd wrote: > Hi! > > > > On 22 September 2013 13:12, Alexander V. Chernikov < > melifaro@yandex-team.ru> wrote: > > >> I'm thinking the same way, but we're stuck with 'forwarding lookup' due >> to problem with egress interface pointer, as I mention earlier. However it >> is interesting to see how much it helps, regardless of locking. >> >> Currently I'm thinking that we should try to change radix to something >> different (it seems that it can be checked fast) and see what happened. >> Luigi's performance numbers for our radix are too awful, and there is a >> patch implementing alternative trie: >> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >> >> > So, I can make educated guesses about why this is better for forwarding > workloads. I'd like to characterize it though. So, what's it doing that's > better? better locking? better caching behaviour? less memory lookups? etc. > > locking affects scalability; but dxr and similar algorithms have much fewer memory lookups, not to mention the huge memory footprint of the freebsd radix tree code. Anyways i'd really encourage you to read the dxr paper, it is short and hopefully can give you a better idea of the details (and with data supporting them) than these short notes. cheers luigi From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 05:34:27 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 86AD75E0; Mon, 23 Sep 2013 05:34:27 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7F52B2D69; Mon, 23 Sep 2013 05:34:26 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id cb5so1783896wib.1 for ; Sun, 22 Sep 2013 22:34:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=WLrObaRrjaudgHrsVVp7UTPOur5ytPU9agZs04Amk5Q=; b=GUI48D0a14dpnmlM+4thtySnazrxLS8QMebZL+zjfrMsxmKy1HOzQauSBn5NNQLWmI PtoJ+78lL+9ewqH/Kr/uW34rbahR3SDTtawk2Y8V6xRJgfwsapM+4jGrFiii2pA4l3So u5kivhOtxUdosaa+lNmz0KMcxiB2RJkYvoeaFhUulfsj9hyAzd2tk1N/Me/3nIONfkE5 XpptXkmG+l9E681dKG0H0dyoO1UeDzkdTe6fcimALMGDyLuJTRWp60bKvCJp8fGSKeaK 8VmaP+IdPQRUsjiiy4iIBbiZluGpsOEGzVAqvYf7ShbDBetKt5LKRNP8nevOvDZhPLPz FzGg== MIME-Version: 1.0 X-Received: by 10.180.9.45 with SMTP id w13mr12257051wia.0.1379914464878; Sun, 22 Sep 2013 22:34:24 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Sun, 22 Sep 2013 22:34:24 -0700 (PDT) In-Reply-To: References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> Date: Sun, 22 Sep 2013 22:34:24 -0700 X-Google-Sender-Auth: ZD6q-sKKWSA9dbPfadU_9a8cm3I Message-ID: Subject: Re: Network stack changes From: Adrian Chadd To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "Alexander V. Chernikov" , Luigi Rizzo , Andre Oppermann , "Andrey V. Elsukov" , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 05:34:27 -0000 On 22 September 2013 21:52, Luigi Rizzo wrote: > locking affects scalability; but dxr and similar algorithms have much fewer > memory lookups, not to mention the huge memory footprint of > the freebsd radix tree code. > > Anyways i'd really encourage you to read the dxr paper, it is short > and hopefully can give you a better idea of the details (and with data > supporting them) than these short notes. > > I read the paper. :-) I believe it! It's not the first paper that I've read that packed a FIB into a sensibly cacheable structure. I'm just as interested however in making sure that we actually give people the tools to inspect this stuff for themselves, rather than all of us hacking up something from scratch every time we want to profile this kind of thing. The other side of this coin is locking, and the paper didn't go into that. Eliminating the radix tree overhead is great; now we just have to avoid grabbing all those locks all the damned time for each frame.. -adrian From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 10:02:51 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 340303A0 for ; Mon, 23 Sep 2013 10:02:51 +0000 (UTC) (envelope-from lists@rewt.org.uk) Received: from hosted.mx.as41113.net (abby.lhr1.as41113.net [91.208.177.20]) by mx1.freebsd.org (Postfix) with ESMTP id EE9E02D29 for ; Mon, 23 Sep 2013 10:02:48 +0000 (UTC) Received: from [192.168.1.218] (unknown [212.9.98.193]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: lists@rewt.org.uk) by hosted.mx.as41113.net (Postfix) with ESMTPSA id 3ck1Ch0czxzCY for ; Mon, 23 Sep 2013 10:57:04 +0100 (BST) Message-ID: <5240106F.6010105@rewt.org.uk> Date: Mon, 23 Sep 2013 10:57:03 +0100 From: Joe Holden User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 10:02:51 -0000 On 23/09/2013 06:34, Adrian Chadd wrote: > On 22 September 2013 21:52, Luigi Rizzo wrote: > > >> locking affects scalability; but dxr and similar algorithms have much fewer >> memory lookups, not to mention the huge memory footprint of >> the freebsd radix tree code. >> >> Anyways i'd really encourage you to read the dxr paper, it is short >> and hopefully can give you a better idea of the details (and with data >> supporting them) than these short notes. >> >> > I read the paper. :-) > > I believe it! It's not the first paper that I've read that packed a FIB > into a sensibly cacheable structure. I'm just as interested however in > making sure that we actually give people the tools to inspect this stuff > for themselves, rather than all of us hacking up something from scratch > every time we want to profile this kind of thing. > > The other side of this coin is locking, and the paper didn't go into that. > Eliminating the radix tree overhead is great; now we just have to avoid > grabbing all those locks all the damned time for each frame.. > > > > -adrian The paper is insane, awesome work - actually faster and more efficient than many popular ASICs.... is there any chance of a patch for HEAD/9 being made? I had a go at forward porting it but there are too many changes that I don't understand :( Cheers, Joe From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 11:06:49 2013 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2A37DC5D for ; Mon, 23 Sep 2013 11:06:49 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1667A2141 for ; Mon, 23 Sep 2013 11:06:49 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r8NB6n5X069532 for ; Mon, 23 Sep 2013 11:06:49 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r8NB6mcG069530 for freebsd-net@FreeBSD.org; Mon, 23 Sep 2013 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 23 Sep 2013 11:06:48 GMT Message-Id: <201309231106.r8NB6mcG069530@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 11:06:49 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/182297 net [cm] ArcNet driver fails to detect the link address - o kern/182212 net [patch] [ng_mppc] ng_mppc(4) blocks on network errors o kern/181970 net [re] LAN RealtekŽ 8111G is not supported by re driver o kern/181931 net [vlan] [lagg] vlan over lagg over mlxen crashes the ke o kern/181823 net [ip6] [patch] make ipv6 mroute return same errror code o kern/181741 net [kernel] [patch] Packet loss when 'control' messages a o kern/181703 net [re] [patch] Fix Realtek 8111G Ethernet controller not o kern/181699 net [ipsec] [patch] IPsec does scale to large SPD / SADB o kern/181657 net [bpf] [patch] BPF_COP/BPF_COPX instruction reservation o kern/181257 net [bge] bge link status change o kern/181236 net [igb] igb driver unstable work o kern/181225 net [infiniband] [patch] unloading ipoib crashes the kerne o kern/181135 net [netmap] [patch] sys/dev/netmap patch for Linux compat o kern/181131 net [netmap] [patch] sys/dev/netmap memory allocation impr o kern/181006 net [run] [patch] mbuf leak in run(4) driver o kern/180893 net [if_ethersubr] [patch] Packets received with own LLADD o kern/180844 net [panic] [re] Intermittent panic (re driver?) o kern/180775 net [bxe] if_bxe driver broken with Broadcom BCM57711 card o kern/180722 net [bluetooth] bluetooth takes 30-50 attempts to pair to s kern/180468 net [request] LOCAL_PEERCRED support for PF_INET o kern/180065 net [netinet6] [patch] Multicast loopback to own host brok o kern/179926 net [lacp] [patch] active aggregator selection bug o kern/179824 net [ixgbe] System (9.1-p4) hangs on heavy ixgbe network t o kern/179733 net [lagg] [patch] interface loses capabilities when proto o kern/179429 net [tap] STP enabled tap bridge o kern/179299 net [igb] Intel X540-T2 - unstable driver a kern/179264 net [vimage] [pf] Core dump with Packet filter and VIMAGE o kern/178947 net [arp] arp rejecting not working o kern/178782 net [ixgbe] 82599EB SFP does not work with passthrough und o kern/178612 net [run] kernel panic due the problems with run driver o kern/178472 net [ip6] [patch] make return code consistent with IPv4 co o kern/178079 net [tcp] Switching TCP CC algorithm panics on sparc64 wit s kern/178071 net FreeBSD unable to recongize Kontron (Industrial Comput o kern/177905 net [xl] [panic] ifmedia_set when pluging CardBus LAN card o kern/177618 net [bridge] Problem with bridge firewall with trunk ports o kern/177417 net [ip6] Invalid protocol value in ipsec6_common_input_cb o kern/177402 net [igb] [pf] problem with ethernet driver igb + pf / alt o kern/177400 net [jme] JMC25x 1000baseT establishment issues o kern/177366 net [ieee80211] negative malloc(9) statistics for 80211nod f kern/177362 net [netinet] [patch] Wrong control used to return TOS o kern/177194 net [netgraph] Unnamed netgraph nodes for vlan interfaces o kern/177139 net [igb] igb drops ethernet ports 2 and 3 o kern/176884 net [re] re0 flapping up/down o kern/176671 net [epair] MAC address for epair device not unique o kern/176484 net [ipsec] [enc] [patch] panic: IPsec + enc(4); device na o kern/176446 net [netinet] [patch] Concurrency in ixgbe driving out-of- o kern/176420 net [kernel] [patch] incorrect errno for LOCAL_PEERCRED o kern/176419 net [kernel] [patch] socketpair support for LOCAL_PEERCRED o kern/176401 net [netgraph] page fault in netgraph o kern/176167 net [ipsec][lagg] using lagg and ipsec causes immediate pa o kern/176097 net [lagg] [patch] lagg/lacp broken when aggregated interf o kern/176027 net [em] [patch] flow control systcl consistency for em dr o kern/176026 net [tcp] [patch] TCP wrappers caused quite a lot of warni o bin/175974 net ppp(8): logic issue o kern/175864 net [re] Intel MB D510MO, onboard ethernet not working aft o kern/175852 net [amd64] [patch] in_cksum_hdr() behaves differently on o kern/175734 net no ethernet detected on system with EG20T PCH chipset o kern/175267 net [pf] [tap] pf + tap keep state problem o kern/175236 net [epair] [gif] epair and gif Devices On Bridge o kern/175182 net [panic] kernel panic on RADIX_MPATH when deleting rout o kern/175153 net [tcp] will there miss a FIN when do TSO? o kern/174959 net [net] [patch] rnh_walktree_from visits spurious nodes o kern/174958 net [net] [patch] rnh_walktree_from makes unreasonable ass o kern/174897 net [route] Interface routes are broken o kern/174851 net [bxe] [patch] UDP checksum offload is wrong in bxe dri o kern/174850 net [bxe] [patch] bxe driver does not receive multicasts o kern/174849 net [bxe] [patch] bxe driver can hang kernel when reset o kern/174822 net [tcp] Page fault in tcp_discardcb under high traffic o kern/174602 net [gif] [ipsec] traceroute issue on gif tunnel with ipse o kern/174535 net [tcp] TCP fast retransmit feature works strange o kern/173871 net [gif] process of 'ifconfig gif0 create hangs' when if_ o kern/173475 net [tun] tun(4) stays opened by PID after process is term o kern/173201 net [ixgbe] [patch] Missing / broken ixgbe sysctl's and tu o kern/173137 net [em] em(4) unable to run at gigabit with 9.1-RC2 o kern/173002 net [patch] data type size problem in if_spppsubr.c o kern/172895 net [ixgb] [ixgbe] do not properly determine link-state o kern/172683 net [ip6] Duplicate IPv6 Link Local Addresses o kern/172675 net [netinet] [patch] sysctl_tcp_hc_list (net.inet.tcp.hos o kern/172113 net [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4 o kern/171840 net [ip6] IPv6 packets transmitting only on queue 0 o kern/171739 net [bce] [panic] bce related kernel panic o kern/171711 net [dummynet] [panic] Kernel panic in dummynet o kern/171532 net [ndis] ndis(4) driver includes 'pccard'-specific code, o kern/171531 net [ndis] undocumented dependency for ndis(4) o kern/171524 net [ipmi] ipmi driver crashes kernel by reboot or shutdow s kern/171508 net [epair] [request] Add the ability to name epair device o kern/171228 net [re] [patch] if_re - eeprom write issues o kern/170701 net [ppp] killl ppp or reboot with active ppp connection c o kern/170267 net [ixgbe] IXGBE_LE32_TO_CPUS is probably an unintentiona o kern/170081 net [fxp] pf/nat/jails not working if checksum offloading o kern/169898 net ifconfig(8) fails to set MTU on multiple interfaces. o kern/169676 net [bge] [hang] system hangs, fully or partially after re o kern/169664 net [bgp] Wrongful replacement of interface connected net o kern/169620 net [ng] [pf] ng_l2tp incoming packet bypass pf firewall o kern/169459 net [ppp] umodem/ppp/3g stopped working after update from o kern/169438 net [ipsec] ipv4-in-ipv6 tunnel mode IPsec does not work p kern/168294 net [ixgbe] [patch] ixgbe driver compiled in kernel has no o kern/168246 net [em] Multiple em(4) not working with qemu o kern/168245 net [arp] [regression] Permanent ARP entry not deleted on o kern/168244 net [arp] [regression] Unable to manually remove permanent o kern/168183 net [bce] bce driver hang system o kern/167947 net [setfib] [patch] arpresolve checks only the default FI o kern/167603 net [ip] IP fragment reassembly's broken: file transfer ov o kern/167500 net [em] [panic] Kernel panics in em driver o kern/167325 net [netinet] [patch] sosend sometimes return EINVAL with o kern/167202 net [igmp]: Sending multiple IGMP packets crashes kernel o kern/166462 net [gre] gre(4) when using a tunnel source address from c o kern/166285 net [arp] FreeBSD v8.1 REL p8 arp: unknown hardware addres o kern/166255 net [net] [patch] It should be possible to disable "promis p kern/165903 net mbuf leak o kern/165622 net [ndis][panic][patch] Unregistered use of FPU in kernel s kern/165562 net [request] add support for Intel i350 in FreeBSD 7.4 o kern/165526 net [bxe] UDP packets checksum calculation whithin if_bxe o kern/165488 net [ppp] [panic] Fatal trap 12 jails and ppp , kernel wit o kern/165305 net [ip6] [request] Feature parity between IP_TOS and IPV6 o kern/165296 net [vlan] [patch] Fix EVL_APPLY_VLID, update EVL_APPLY_PR o kern/165181 net [igb] igb freezes after about 2 weeks of uptime o kern/165174 net [patch] [tap] allow tap(4) to keep its address on clos o kern/165152 net [ip6] Does not work through the issue of ipv6 addresse o kern/164495 net [igb] connect double head igb to switch cause system t o kern/164490 net [pfil] Incorrect IP checksum on pfil pass from ip_outp o kern/164475 net [gre] gre misses RUNNING flag after a reboot o kern/164265 net [netinet] [patch] tcp_lro_rx computes wrong checksum i o kern/163903 net [igb] "igb0:tx(0)","bpf interface lock" v2.2.5 9-STABL o kern/163481 net freebsd do not add itself to ping route packet o kern/162927 net [tun] Modem-PPP error ppp[1538]: tun0: Phase: Clearing o kern/162558 net [dummynet] [panic] seldom dummynet panics o kern/162153 net [em] intel em driver 7.2.4 don't compile o kern/162110 net [igb] [panic] RELENG_9 panics on boot in IGB driver - o kern/162028 net [ixgbe] [patch] misplaced #endif in ixgbe.c o kern/161277 net [em] [patch] BMC cannot receive IPMI traffic after loa o kern/160873 net [igb] igb(4) from HEAD fails to build on 7-STABLE o kern/160750 net Intel PRO/1000 connection breaks under load until rebo o kern/160693 net [gif] [em] Multicast packet are not passed from GIF0 t o kern/160293 net [ieee80211] ppanic] kernel panic during network setup o kern/160206 net [gif] gifX stops working after a while (IPv6 tunnel) o kern/159817 net [udp] write UDPv4: No buffer space available (code=55) o kern/159629 net [ipsec] [panic] kernel panic with IPsec in transport m o kern/159621 net [tcp] [panic] panic: soabort: so_count o kern/159603 net [netinet] [patch] in_ifscrubprefix() - network route c o kern/159601 net [netinet] [patch] in_scrubprefix() - loopback route re o kern/159294 net [em] em watchdog timeouts o kern/159203 net [wpi] Intel 3945ABG Wireless LAN not support IBSS o kern/158930 net [bpf] BPF element leak in ifp->bpf_if->bif_dlist o kern/158726 net [ip6] [patch] ICMPv6 Router Announcement flooding limi o kern/158694 net [ix] [lagg] ix0 is not working within lagg(4) o kern/158665 net [ip6] [panic] kernel pagefault in in6_setscope() o kern/158635 net [em] TSO breaks BPF packet captures with em driver f kern/157802 net [dummynet] [panic] kernel panic in dummynet o kern/157785 net amd64 + jail + ipfw + natd = very slow outbound traffi o kern/157418 net [em] em driver lockup during boot on Supermicro X9SCM- o kern/157410 net [ip6] IPv6 Router Advertisements Cause Excessive CPU U o kern/157287 net [re] [panic] INVARIANTS panic (Memory modified after f o kern/157200 net [network.subr] [patch] stf(4) can not communicate betw o kern/157182 net [lagg] lagg interface not working together with epair o kern/156877 net [dummynet] [panic] dummynet move_pkt() null ptr derefe o kern/156667 net [em] em0 fails to init on CURRENT after March 17 o kern/156408 net [vlan] Routing failure when using VLANs vs. Physical e o kern/156328 net [icmp]: host can ping other subnet but no have IP from o kern/156317 net [ip6] Wrong order of IPv6 NS DAD/MLD Report o kern/156283 net [ip6] [patch] nd6_ns_input - rtalloc_mpath does not re o kern/156279 net [if_bridge][divert][ipfw] unable to correctly re-injec o kern/156226 net [lagg]: failover does not announce the failover to swi o kern/156030 net [ip6] [panic] Crash in nd6_dad_start() due to null ptr o kern/155680 net [multicast] problems with multicast s kern/155642 net [new driver] [request] Add driver for Realtek RTL8191S o kern/155597 net [panic] Kernel panics with "sbdrop" message o kern/155420 net [vlan] adding vlan break existent vlan o kern/155177 net [route] [panic] Panic when inject routes in kernel o kern/155010 net [msk] ntfs-3g via iscsi using msk driver cause kernel o kern/154943 net [gif] ifconfig gifX create on existing gifX clears IP s kern/154851 net [new driver] [request]: Port brcm80211 driver from Lin o kern/154850 net [netgraph] [patch] ng_ether fails to name nodes when t o kern/154679 net [em] Fatal trap 12: "em1 taskq" only at startup (8.1-R o kern/154600 net [tcp] [panic] Random kernel panics on tcp_output o kern/154557 net [tcp] Freeze tcp-session of the clients, if in the gat o kern/154443 net [if_bridge] Kernel module bridgestp.ko missing after u o kern/154286 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/154255 net [nfs] NFS not responding o kern/154214 net [stf] [panic] Panic when creating stf interface o kern/154185 net race condition in mb_dupcl p kern/154169 net [multicast] [ip6] Node Information Query multicast add o kern/154134 net [ip6] stuck kernel state in LISTEN on ipv6 daemon whic o kern/154091 net [netgraph] [panic] netgraph, unaligned mbuf? o conf/154062 net [vlan] [patch] change to way of auto-generatation of v o kern/153937 net [ral] ralink panics the system (amd64 freeBSDD 8.X) wh o kern/153936 net [ixgbe] [patch] MPRC workaround incorrectly applied to o kern/153816 net [ixgbe] ixgbe doesn't work properly with the Intel 10g o kern/153772 net [ixgbe] [patch] sysctls reference wrong XON/XOFF varia o kern/153497 net [netgraph] netgraph panic due to race conditions o kern/153454 net [patch] [wlan] [urtw] Support ad-hoc and hostap modes o kern/153308 net [em] em interface use 100% cpu o kern/153244 net [em] em(4) fails to send UDP to port 0xffff o kern/152893 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/152853 net [em] tftpd (and likely other udp traffic) fails over e o kern/152828 net [em] poor performance on 8.1, 8.2-PRE o kern/152569 net [net]: Multiple ppp connections and routing table prob o kern/152235 net [arp] Permanent local ARP entries are not properly upd o kern/152141 net [vlan] [patch] encapsulate vlan in ng_ether before out o kern/152036 net [libc] getifaddrs(3) returns truncated sockaddrs for n o kern/151690 net [ep] network connectivity won't work until dhclient is o kern/151681 net [nfs] NFS mount via IPv6 leads to hang on client with o kern/151593 net [igb] [panic] Kernel panic when bringing up igb networ o kern/150920 net [ixgbe][igb] Panic when packets are dropped with heade o kern/150557 net [igb] igb0: Watchdog timeout -- resetting o kern/150251 net [patch] [ixgbe] Late cable insertion broken o kern/150249 net [ixgbe] Media type detection broken o bin/150224 net ppp(8) does not reassign static IP after kill -KILL co f kern/149969 net [wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149643 net [rum] device not sending proper beacon frames in ap mo o kern/149609 net [panic] reboot after adding second default route o kern/149117 net [inet] [patch] in_pcbbind: redundant test o kern/149086 net [multicast] Generic multicast join failure in 8.1 o kern/148018 net [flowtable] flowtable crashes on ia64 o kern/147912 net [boot] FreeBSD 8 Beta won't boot on Thinkpad i1300 11 o kern/147894 net [ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147155 net [ip6] setfb not work with ipv6 o kern/146845 net [libc] close(2) returns error 54 (connection reset by f kern/146792 net [flowtable] flowcleaner 100% cpu's core load o kern/146719 net [pf] [panic] PF or dumynet kernel panic o kern/146534 net [icmp6] wrong source address in echo reply o kern/146427 net [mwl] Additional virtual access points don't work on m f kern/146394 net [vlan] IP source address for outgoing connections o bin/146377 net [ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net [vlan] wrong destination MAC address o kern/146165 net [wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net [ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net [panic] mpd + CoA = kernel panic o kern/145825 net [panic] panic: soabort: so_count o kern/145728 net [lagg] Stops working lagg between two servers. p kern/145600 net TCP/ECN behaves different to CE/CWR than ns2 reference f kern/144917 net [flowtable] [panic] flowtable crashes system [regressi o kern/144882 net MacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net [if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net [rc.d] async dhclient breaks stuff for too many people o kern/144616 net [nat] [panic] ip_nat panic FreeBSD 7.2 f kern/144315 net [ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/144231 net bind/connect/sendto too strict about sockaddr length o kern/143846 net [gif] bringing gif3 tunnel down causes gif0 tunnel to s kern/143673 net [stf] [request] there should be a way to support multi s kern/143666 net [ip6] [request] PMTU black hole detection not implemen o kern/143622 net [pfil] [patch] unlock pfil lock while calling firewall o kern/143593 net [ipsec] When using IPSec, tcpdump doesn't show outgoin o kern/143591 net [ral] RT2561C-based DLink card (DWL-510) fails to work o kern/143208 net [ipsec] [gif] IPSec over gif interface not working o kern/143034 net [panic] system reboots itself in tcp code [regression] o kern/142877 net [hang] network-related repeatable 8.0-STABLE hard hang o kern/142774 net Problem with outgoing connections on interface with mu o kern/142772 net [libc] lla_lookup: new lle malloc failed f kern/142518 net [em] [lagg] Problem on 8.0-STABLE with em and lagg o kern/142018 net [iwi] [patch] Possibly wrong interpretation of beacon- o kern/141861 net [wi] data garbled with WEP and wi(4) with Prism 2.5 f kern/141741 net Etherlink III NIC won't work after upgrade to FBSD 8, o kern/140742 net rum(4) Two asus-WL167G adapters cannot talk to each ot o kern/140682 net [netgraph] [panic] random panic in netgraph f kern/140634 net [vlan] destroying if_lagg interface with if_vlan membe o kern/140619 net [ifnet] [patch] refine obsolete if_var.h comments desc o kern/140346 net [wlan] High bandwidth use causes loss of wlan connecti o kern/140142 net [ip6] [panic] FreeBSD 7.2-amd64 panic w/IPv6 o kern/140066 net [bwi] install report for 8.0 RC 2 (multiple problems) o kern/139387 net [ipsec] Wrong lenth of PF_KEY messages in promiscuous o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL p kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR if_adata/ o kern/138266 net [panic] kernel panic when udp benchmark test used as r f kern/138029 net [bpf] [panic] periodically kernel panic and reboot o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 p bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o conf/132851 net [patch] rc.conf(5): allow to setfib(1) for service run o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o bin/131365 net route(8): route add changes interpretation of network f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129517 net [ipsec] [panic] double fault / stack overflow f kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o bin/128954 net ifconfig(8) deletes valid routes o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by p kern/127360 net [socket] TOE socket options missing from sosetopt() o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126339 net [ipw] ipw driver drops the connection o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123758 net [panic] panic while restarting net/freenet6 o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices f kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122252 net [ipmi] [bge] IPMI problem with BCM5704 (does not work o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup ieee o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121534 net [ipl] [nat] FreeBSD Release 6.3 Kernel Trap 12: o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] [security] ppp(8): fix local stack overflow in o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/120966 net [rum] kernel panic with if_rum and WPA encryption o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/118727 net [netgraph] [patch] [request] add new ng_pf module o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111537 net [inet6] [patch] ip6_input() treats mbuf cluster wrong o kern/111457 net [ral] ral(4) freeze o kern/110284 net [if_ethersubr] Invalid Assumption in SIOCSIFADDR in et o kern/110249 net [kernel] [regression] [patch] setsockopt() error regre o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] f kern/108197 net [panic] [gif] [ip6] if_delmulti reference counting pan o kern/107944 net [wi] [patch] Forget to unlock mutex-locks o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106444 net [netgraph] [panic] Kernel Panic on Binding to an ip to o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/104738 net [inet] [patch] Reentrant problem with inet_ntoa in the o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o kern/102540 net [netgraph] [patch] supporting vlan(4) by ng_fec(4) o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working o kern/97306 net [netgraph] NG_L2TP locks after connection with failed o conf/97014 net [gif] gifconfig_gif? in rc.conf does not recognize IPv f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging o kern/87421 net [netgraph] [panic]: ng_ether + ng_eiface + if_bridge o kern/86871 net [tcp] [patch] allocation logic for PCBs in TIME_WAIT s o kern/86427 net [lor] Deadlock with FASTIPSEC and nat o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ o bin/82975 net route change does not parse classfull network as given o kern/82881 net [netgraph] [panic] ng_fec(4) causes kernel panic after o kern/82468 net Using 64MB tcp send/recv buffers, trafficflow stops, i o bin/82185 net [patch] ndp(8) can delete the incorrect entry o kern/81095 net IPsec connection stops working if associated network i o kern/78968 net FreeBSD freezes on mbufs exhaustion (network interface o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o kern/77341 net [ip6] problems with IPV6 implementation o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time a kern/71474 net [route] route lookup does not skip interfaces marked d o kern/71469 net default route to internet magically disappears with mu o kern/68889 net [panic] m_copym, length > size of mbuf chain o kern/66225 net [netgraph] [patch] extend ng_eiface(4) control message o kern/65616 net IPSEC can't detunnel GRE packets after real ESP encryp s kern/60293 net [patch] FreeBSD arp poison patch a kern/56233 net IPsec tunnel (ESP) over IPv6: MTU computation is wrong s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr o kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/31940 net ip queue length too short for >500kpps o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna f kern/24959 net [patch] proper TCP_NOPUSH/TCP_CORK compatibility o conf/23063 net [arp] [patch] for static ARP tables in rc.network o kern/21998 net [socket] [patch] ident only for outgoing connections o kern/5877 net [socket] sb_cc counts control data as well as data dat 469 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 11:37:46 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2200466A; Mon, 23 Sep 2013 11:37:46 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id D0D8624F2; Mon, 23 Sep 2013 11:37:45 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1VO4UW-0001lI-DE; Mon, 23 Sep 2013 15:39:48 +0400 Date: Mon, 23 Sep 2013 15:39:48 +0400 From: Slawa Olhovchenkov To: "Alexander V. Chernikov" Subject: Re: Network stack changes Message-ID: <20130923113948.GA5647@zxy.spb.ru> References: <521E41CB.30700@yandex-team.ru> <521E78B0.6080709@freebsd.org> <523F4BED.6000804@yandex-team.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <523F4BED.6000804@yandex-team.ru> User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: adrian@freebsd.org, Andre Oppermann , freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, luigi@freebsd.org, ae@FreeBSD.org, FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 11:37:46 -0000 On Sun, Sep 22, 2013 at 11:58:37PM +0400, Alexander V. Chernikov wrote: > I've found the paper I was talking about: > http://info.iet.unipi.it/~luigi/papers/20120601-dxr.pdf > > It claims that our radix is able to do 6MPPS/core and it does not scale > with number of cores. Our radix is bugly and don't work corretly. From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 11:50:16 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5B08AE69 for ; Mon, 23 Sep 2013 11:50:16 +0000 (UTC) (envelope-from yar.tikhiy@gmail.com) Received: from mail-ve0-x232.google.com (mail-ve0-x232.google.com [IPv6:2607:f8b0:400c:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1FD8025BE for ; Mon, 23 Sep 2013 11:50:16 +0000 (UTC) Received: by mail-ve0-f178.google.com with SMTP id jw12so2302243veb.9 for ; Mon, 23 Sep 2013 04:50:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=woOUzKWkwR2z+vZRoET45pOmB+HuP+y+YHTVlTz1q9w=; b=pquGc9GnC2gKyOp5jX8LGzX93TxK/5MWu/hwh7t9bFTFUHtePcsUNbLwfhNdYPc4j4 AjVoNymzwLiwX4tEYiCcRsD/izMtKbNJCxLtWtTNoV25LDAa9rzi2RV6pclQnGmptvk3 Wo89r8v1v9RcxiElcFr1S0BfSCXLi226WlyptVMzihtma95CFiwPddsqH42Pa6w+e6bi XfNsy/zZvSIT53qXhLi/vkLdHQEDVCLh+FmOCrpnp6KWyFdA4vO96/l+4qg2XPjPuZDt KhiMm8+yI4vqb29PlrtEJFcz3AUKCncZHlxiELnTc4c28r8Ux00tiCn6TI4nxZ4wxrZG RVOw== MIME-Version: 1.0 X-Received: by 10.220.237.208 with SMTP id kp16mr21738869vcb.4.1379937015245; Mon, 23 Sep 2013 04:50:15 -0700 (PDT) Received: by 10.58.90.68 with HTTP; Mon, 23 Sep 2013 04:50:15 -0700 (PDT) Date: Mon, 23 Sep 2013 21:50:15 +1000 Message-ID: Subject: Free book draft: IPv6 for IPv4 Experts From: Yar Tikhiy To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 11:50:16 -0000 Hi all, Not a complete stranger to the project, I hope the following 'advertisement' will be appropriate here. When sometime around 2007 I finally had to make real sense of IPv6, I found a magic garden where brilliant ideas bloomed, and I felt it would be a shame to have all that beauty only to myself. I simply couldn't resist trying to structure those ideas up so that each of them shined in a crown of logical connections with others and with facts an average IPv4 wizard could be expected to know. Then it took some slow-paced rethinking, editing, and formatting to arrive a few years later at what can be considered a textbook intended for the battle-hardened veterans of the IPv4 era who are sufficiently curious and confident at the same time to stretch their current knowledge to the limit in order to discover the whys and wherefores of the IPv6 architecture instead of just dull facts about it such as the IPv6 address length. (As an exercise, try to tell in under 3 seconds what the famous length equals to _in bytes_.) Unsure if I want to get into the publishing hassle, I figured the best way to launch my work into the interhuman information space would be just to offer the final draft to the attention of the community. If you like it, the biggest favor you can do me will be to share it via your social network of choice. If you don't quite like it---well, then you probably don't have to share it. :-) The project page is: https://sites.google.com/site/yartikhiy/home/ipv6book An e-reader friendly PDF as well as a conventional A4 size PDF is available. Hoping you will enjoy the reading as much as I have enjoyed the writing. Cheers, Yar (formerly yar@) From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 14:18:39 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0DC9D233 for ; Mon, 23 Sep 2013 14:18:39 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CC2F220BD for ; Mon, 23 Sep 2013 14:18:38 +0000 (UTC) Received: from [192.168.43.26] (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.7/8.14.7) with ESMTP id r8NEIc6q069031; Mon, 23 Sep 2013 10:18:38 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <52404DA7.1040400@sentex.net> Date: Mon, 23 Sep 2013 10:18:15 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Yar Tikhiy Subject: Re: Free book draft: IPv6 for IPv4 Experts References: In-Reply-To: X-Enigmail-Version: 1.4.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.74 on 64.7.153.18 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 14:18:39 -0000 On 9/23/2013 7:50 AM, Yar Tikhiy wrote: > > The project page is: https://sites.google.com/site/yartikhiy/home/ipv6book > > An e-reader friendly PDF as well as a conventional A4 size PDF is available. > > Hoping you will enjoy the reading as much as I have enjoyed the writing. Wow! I just had a look at the TOC and it looks like a great addition to the spare resources that are out there. I will certainly have a look through it in the coming days. Thanks for sharing with the community! ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 18:10:01 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D83EF89F for ; Mon, 23 Sep 2013 18:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A99BA2EEF for ; Mon, 23 Sep 2013 18:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r8NIA0mt064335 for ; Mon, 23 Sep 2013 18:10:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r8NIA01m064334; Mon, 23 Sep 2013 18:10:00 GMT (envelope-from gnats) Date: Mon, 23 Sep 2013 18:10:00 GMT Message-Id: <201309231810.r8NIA01m064334@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Gabor Berczi Subject: Re: kern/182297: [cm] ArcNet driver fails to detect the link address - and does not work at all X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Gabor Berczi List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 18:10:01 -0000 The following reply was made to PR kern/182297; it has been noted by GNATS. From: Gabor Berczi To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/182297: [cm] ArcNet driver fails to detect the link address - and does not work at all Date: Mon, 23 Sep 2013 19:39:38 +0200 Tried older kernels, the last release from the v5.x tree was broken too. I can provide a test environment. Compensation will be given for the fix, if that's an incentive. From owner-freebsd-net@FreeBSD.ORG Mon Sep 23 22:46:47 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8EB6AE72; Mon, 23 Sep 2013 22:46:47 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4269C2F8F; Mon, 23 Sep 2013 22:46:47 +0000 (UTC) Received: by mail-pd0-f170.google.com with SMTP id x10so3809582pdj.29 for ; Mon, 23 Sep 2013 15:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UjilBwU9OcaYlukiK4g1agmh0oG9lXC8IZz6YH7pBlg=; b=zNd2nK4fBP8S1qNkzhLLiAaFq3C4x9cs2l/LTZGD8uliIScxephPgtW+7SCmsbDgyP zPOwbSas1L4JqJYHFeXl+xDyHGgcKGQxW90RfR95vFmZzoqsZPL6IkW4nE98w++cfJsZ +e6CJwB7dPIYAg0V1byVyhshHzsVmidAdwxDXn3Aij1tX5RjR+b7zexAkbHtjh1aDB/L FafyXOjh/rDms1JVnjmCUv9t2V6LedW+gx722WWwMGk7TiOi77IdH6DE5B9lYEfwZ4g7 trkHotGxZ1WWlEq6SF2TtwsgUIzr8ghc2hO73/neyiPvynvGoPaLlM/NHjuRcQjZss2Q lv5A== MIME-Version: 1.0 X-Received: by 10.68.11.103 with SMTP id p7mr3431565pbb.84.1379976406783; Mon, 23 Sep 2013 15:46:46 -0700 (PDT) Received: by 10.70.30.98 with HTTP; Mon, 23 Sep 2013 15:46:46 -0700 (PDT) In-Reply-To: <523F4F14.9090404@yandex-team.ru> References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> Date: Tue, 24 Sep 2013 01:46:46 +0300 Message-ID: Subject: Re: Network stack changes From: Sami Halabi To: "Alexander V. Chernikov" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Adrian Chadd , Andre Oppermann , "freebsd-hackers@freebsd.org" , "freebsd-arch@freebsd.org" , Luigi Rizzo , "Andrey V. Elsukov" , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2013 22:46:47 -0000 Hi, > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff I've tried the diff in 10-current, applied cleanly but had errors compiling new kernel... is there any work to make it work? i'd love to test it. Sami On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < melifaro@yandex-team.ru> wrote: > On 29.08.2013 15:49, Adrian Chadd wrote: > >> Hi, >> > Hello Adrian! > I'm very sorry for the looong reply. > > > >> There's a lot of good stuff to review here, thanks! >> >> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to >> keep locking things like that on a per-packet basis. We should be able to >> do this in a cleaner way - we can defer RX into a CPU pinned taskqueue and >> convert the interrupt handler to a fast handler that just schedules that >> taskqueue. We can ignore the ithread entirely here. >> >> What do you think? >> > Well, it sounds good :) But performance numbers and Jack opinion is more > important :) > > Are you going to Malta? > > >> Totally pie in the sky handwaving at this point: >> >> * create an array of mbuf pointers for completed mbufs; >> * populate the mbuf array; >> * pass the array up to ether_demux(). >> >> For vlan handling, it may end up populating its own list of mbufs to push >> up to ether_demux(). So maybe we should extend the API to have a bitmap of >> packets to actually handle from the array, so we can pass up a larger array >> of mbufs, note which ones are for the destination and then the upcall can >> mark which frames its consumed. >> >> I specifically wonder how much work/benefit we may see by doing: >> >> * batching packets into lists so various steps can batch process things >> rather than run to completion; >> * batching the processing of a list of frames under a single lock >> instance - eg, if the forwarding code could do the forwarding lookup for >> 'n' packets under a single lock, then pass that list of frames up to >> inet_pfil_hook() to do the work under one lock, etc, etc. >> > I'm thinking the same way, but we're stuck with 'forwarding lookup' due to > problem with egress interface pointer, as I mention earlier. However it is > interesting to see how much it helps, regardless of locking. > > Currently I'm thinking that we should try to change radix to something > different (it seems that it can be checked fast) and see what happened. > Luigi's performance numbers for our radix are too awful, and there is a > patch implementing alternative trie: > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff > > > > >> Here, the processing would look less like "grab lock and process to >> completion" and more like "mark and sweep" - ie, we have a list of frames >> that we mark as needing processing and mark as having been processed at >> each layer, so we know where to next dispatch them. >> >> I still have some tool coding to do with PMC before I even think about >> tinkering with this as I'd like to measure stuff like per-packet latency as >> well as top-level processing overhead (ie, CPU_CLK_UNHALTED.THREAD_P / >> lagg0 TX bytes/pkts, RX bytes/pkts, NIC interrupts on that core, etc.) >> > That will be great to see! > >> >> Thanks, >> >> >> >> -adrian >> >> > ______________________________**_________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org > " > -- Sami Halabi Information Systems Engineer NMS Projects Expert FreeBSD SysAdmin Expert On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < melifaro@yandex-team.ru> wrote: > On 29.08.2013 15:49, Adrian Chadd wrote: > >> Hi, >> > Hello Adrian! > I'm very sorry for the looong reply. > > > >> There's a lot of good stuff to review here, thanks! >> >> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to >> keep locking things like that on a per-packet basis. We should be able to >> do this in a cleaner way - we can defer RX into a CPU pinned taskqueue and >> convert the interrupt handler to a fast handler that just schedules that >> taskqueue. We can ignore the ithread entirely here. >> >> What do you think? >> > Well, it sounds good :) But performance numbers and Jack opinion is more > important :) > > Are you going to Malta? > > >> Totally pie in the sky handwaving at this point: >> >> * create an array of mbuf pointers for completed mbufs; >> * populate the mbuf array; >> * pass the array up to ether_demux(). >> >> For vlan handling, it may end up populating its own list of mbufs to push >> up to ether_demux(). So maybe we should extend the API to have a bitmap of >> packets to actually handle from the array, so we can pass up a larger array >> of mbufs, note which ones are for the destination and then the upcall can >> mark which frames its consumed. >> >> I specifically wonder how much work/benefit we may see by doing: >> >> * batching packets into lists so various steps can batch process things >> rather than run to completion; >> * batching the processing of a list of frames under a single lock >> instance - eg, if the forwarding code could do the forwarding lookup for >> 'n' packets under a single lock, then pass that list of frames up to >> inet_pfil_hook() to do the work under one lock, etc, etc. >> > I'm thinking the same way, but we're stuck with 'forwarding lookup' due to > problem with egress interface pointer, as I mention earlier. However it is > interesting to see how much it helps, regardless of locking. > > Currently I'm thinking that we should try to change radix to something > different (it seems that it can be checked fast) and see what happened. > Luigi's performance numbers for our radix are too awful, and there is a > patch implementing alternative trie: > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff > > > > >> Here, the processing would look less like "grab lock and process to >> completion" and more like "mark and sweep" - ie, we have a list of frames >> that we mark as needing processing and mark as having been processed at >> each layer, so we know where to next dispatch them. >> >> I still have some tool coding to do with PMC before I even think about >> tinkering with this as I'd like to measure stuff like per-packet latency as >> well as top-level processing overhead (ie, CPU_CLK_UNHALTED.THREAD_P / >> lagg0 TX bytes/pkts, RX bytes/pkts, NIC interrupts on that core, etc.) >> > That will be great to see! > >> >> Thanks, >> >> >> >> -adrian >> >> > ______________________________**_________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org > " > -- Sami Halabi Information Systems Engineer NMS Projects Expert FreeBSD SysAdmin Expert From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 07:58:31 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2E99BDED; Tue, 24 Sep 2013 07:58:31 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5411428EC; Tue, 24 Sep 2013 07:58:29 +0000 (UTC) Received: from dyn10.nxlab.fer.hr (161.53.63.210) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Tue, 24 Sep 2013 09:57:14 +0200 From: Marko Zec To: Subject: Re: Network stack changes Date: Tue, 24 Sep 2013 09:58:05 +0200 User-Agent: KMail/1.9.10 References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201309240958.06172.zec@fer.hr> X-Originating-IP: [161.53.63.210] Cc: "Alexander V. Chernikov" , Adrian Chadd , Andre Oppermann , FreeBSD Net , Luigi Rizzo , "Andrey V. Elsukov" , "freebsd-arch@freebsd.org" , Sami Halabi X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 07:58:31 -0000 On Tuesday 24 September 2013 00:46:46 Sami Halabi wrote: > Hi, > > > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >er.hr/dxr/stable_8_20120824.diff> > > I've tried the diff in 10-current, applied cleanly but had errors > compiling new kernel... is there any work to make it work? i'd love to > test it. Even if you'd make it compile on current, you could only run synthetic tests measuring lookup performance using streams of random keys, as outlined in the paper (btw. the paper at Luigi's site is an older draft, the final version with slightly revised benchmarks is available here: http://www.sigcomm.org/sites/default/files/ccr/papers/2012/October/2378956-2378961.pdf) I.e. the code only hooks into the routing API for testing purposes, but is completely disconnected from the forwarding path. We have a prototype in the works which combines DXR with Netmap in userspace and is capable of sustaining well above line rate forwarding with full-sized BGP views using Intel 10G cards on commodity multicore machines. The work was somewhat stalled during the summer but I plan to wrap it up and release the code until the end of this year. With recent advances in netmap it might also be feasible to merge DXR and netmap entirely inside the kernel but I've not explored that path yet... Marko > Sami > > > On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > > melifaro@yandex-team.ru> wrote: > > On 29.08.2013 15:49, Adrian Chadd wrote: > >> Hi, > > > > Hello Adrian! > > I'm very sorry for the looong reply. > > > >> There's a lot of good stuff to review here, thanks! > >> > >> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to > >> keep locking things like that on a per-packet basis. We should be able > >> to do this in a cleaner way - we can defer RX into a CPU pinned > >> taskqueue and convert the interrupt handler to a fast handler that > >> just schedules that taskqueue. We can ignore the ithread entirely > >> here. > >> > >> What do you think? > > > > Well, it sounds good :) But performance numbers and Jack opinion is > > more important :) > > > > Are you going to Malta? > > > >> Totally pie in the sky handwaving at this point: > >> > >> * create an array of mbuf pointers for completed mbufs; > >> * populate the mbuf array; > >> * pass the array up to ether_demux(). > >> > >> For vlan handling, it may end up populating its own list of mbufs to > >> push up to ether_demux(). So maybe we should extend the API to have a > >> bitmap of packets to actually handle from the array, so we can pass up > >> a larger array of mbufs, note which ones are for the destination and > >> then the upcall can mark which frames its consumed. > >> > >> I specifically wonder how much work/benefit we may see by doing: > >> > >> * batching packets into lists so various steps can batch process > >> things rather than run to completion; > >> * batching the processing of a list of frames under a single lock > >> instance - eg, if the forwarding code could do the forwarding lookup > >> for 'n' packets under a single lock, then pass that list of frames up > >> to inet_pfil_hook() to do the work under one lock, etc, etc. > > > > I'm thinking the same way, but we're stuck with 'forwarding lookup' due > > to problem with egress interface pointer, as I mention earlier. However > > it is interesting to see how much it helps, regardless of locking. > > > > Currently I'm thinking that we should try to change radix to something > > different (it seems that it can be checked fast) and see what happened. > > Luigi's performance numbers for our radix are too awful, and there is a > > patch implementing alternative trie: > > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >er.hr/dxr/stable_8_20120824.diff> > > > >> Here, the processing would look less like "grab lock and process to > >> completion" and more like "mark and sweep" - ie, we have a list of > >> frames that we mark as needing processing and mark as having been > >> processed at each layer, so we know where to next dispatch them. > >> > >> I still have some tool coding to do with PMC before I even think about > >> tinkering with this as I'd like to measure stuff like per-packet > >> latency as well as top-level processing overhead (ie, > >> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC > >> interrupts on that core, etc.) > > > > That will be great to see! > > > >> Thanks, > >> > >> > >> > >> -adrian > > > > ______________________________**_________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/**mailman/listinfo/freebsd-net >eebsd.org/mailman/listinfo/freebsd-net> To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@**freebsd.org >org> " > > -- > Sami Halabi > Information Systems Engineer > NMS Projects Expert > FreeBSD SysAdmin Expert > > > On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > > melifaro@yandex-team.ru> wrote: > > On 29.08.2013 15:49, Adrian Chadd wrote: > >> Hi, > > > > Hello Adrian! > > I'm very sorry for the looong reply. > > > >> There's a lot of good stuff to review here, thanks! > >> > >> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to > >> keep locking things like that on a per-packet basis. We should be able > >> to do this in a cleaner way - we can defer RX into a CPU pinned > >> taskqueue and convert the interrupt handler to a fast handler that > >> just schedules that taskqueue. We can ignore the ithread entirely > >> here. > >> > >> What do you think? > > > > Well, it sounds good :) But performance numbers and Jack opinion is > > more important :) > > > > Are you going to Malta? > > > >> Totally pie in the sky handwaving at this point: > >> > >> * create an array of mbuf pointers for completed mbufs; > >> * populate the mbuf array; > >> * pass the array up to ether_demux(). > >> > >> For vlan handling, it may end up populating its own list of mbufs to > >> push up to ether_demux(). So maybe we should extend the API to have a > >> bitmap of packets to actually handle from the array, so we can pass up > >> a larger array of mbufs, note which ones are for the destination and > >> then the upcall can mark which frames its consumed. > >> > >> I specifically wonder how much work/benefit we may see by doing: > >> > >> * batching packets into lists so various steps can batch process > >> things rather than run to completion; > >> * batching the processing of a list of frames under a single lock > >> instance - eg, if the forwarding code could do the forwarding lookup > >> for 'n' packets under a single lock, then pass that list of frames up > >> to inet_pfil_hook() to do the work under one lock, etc, etc. > > > > I'm thinking the same way, but we're stuck with 'forwarding lookup' due > > to problem with egress interface pointer, as I mention earlier. However > > it is interesting to see how much it helps, regardless of locking. > > > > Currently I'm thinking that we should try to change radix to something > > different (it seems that it can be checked fast) and see what happened. > > Luigi's performance numbers for our radix are too awful, and there is a > > patch implementing alternative trie: > > http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >er.hr/dxr/stable_8_20120824.diff> > > > >> Here, the processing would look less like "grab lock and process to > >> completion" and more like "mark and sweep" - ie, we have a list of > >> frames that we mark as needing processing and mark as having been > >> processed at each layer, so we know where to next dispatch them. > >> > >> I still have some tool coding to do with PMC before I even think about > >> tinkering with this as I'd like to measure stuff like per-packet > >> latency as well as top-level processing overhead (ie, > >> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC > >> interrupts on that core, etc.) > > > > That will be great to see! > > > >> Thanks, > >> > >> > >> > >> -adrian > > > > ______________________________**_________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/**mailman/listinfo/freebsd-net >eebsd.org/mailman/listinfo/freebsd-net> To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@**freebsd.org >org> " From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 08:16:59 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id ECC22442 for ; Tue, 24 Sep 2013 08:16:59 +0000 (UTC) (envelope-from adpacifyer@gmail.com) Received: from mail-we0-x22f.google.com (mail-we0-x22f.google.com [IPv6:2a00:1450:400c:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 549B12A10 for ; Tue, 24 Sep 2013 08:16:59 +0000 (UTC) Received: by mail-we0-f175.google.com with SMTP id q59so4079637wes.6 for ; Tue, 24 Sep 2013 01:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ey3Tqlx8a+wJC0m51dBU3ujD0HkE+xLMiHT/GYzDPnI=; b=utU/bUT8SKLrAYNtX5EDWieRJzQeax9796HsPgLU3SuOR4q3mLKMmOLjDd5SSUAJuS BkDCTWKLiHok8pIyUzH6Qlt2EVrI9jedWcz9GcZfjMEJC+AQgYqq5K2ACKhXWrfqy0Ar fAZpBHncheWo1UZKXqBRQjXQhxsxOk5TSlims/Vcs+yG8spbCN197BuN9GfvaKx93TXi WjTwZ6WEwIBcB2jQSOAZWG7oStcEjdF+M8HRs7fz4gYopLnnwMzvzelJW9szvTN7Bir4 5bGnUvDM6fLixmOA/+Op6yZi6+a3X9UNXXQocEf4nk9I+SyXnDVCYcbz/SUMfIMFGS/3 s6oA== MIME-Version: 1.0 X-Received: by 10.180.98.105 with SMTP id eh9mr16953941wib.56.1380010617725; Tue, 24 Sep 2013 01:16:57 -0700 (PDT) Received: by 10.217.113.72 with HTTP; Tue, 24 Sep 2013 01:16:57 -0700 (PDT) Date: Tue, 24 Sep 2013 11:16:57 +0300 Message-ID: Subject: FreeBSD 9 em driver high inerrupt load + network crashes From: Oleg Tarasov To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 08:17:00 -0000 Hello, I am currently experiencing a problem on one of my servers. Interestingly it is the only one with such problem however it is the most loaded one. Sometimes on this server network becomes completely unresponsitive forcing me to attach console and reboot the system. There are no records in the logs that could possibly point me to any direction. However I am seing unexpected CPU interrupt load on this server: CPU: 0.7% user, 0.0% nice, 0.2% system, 18.1% interrupt, 80.9% idle # vmstat -i interrupt total rate irq16: em0 aac0+ 8776771724 99009 <--- This is definitely a problem irq23: ehci1 177381 2 cpu0:timer 8442700 95 irq264: em1:rx 0 12525811 141 irq265: em1:tx 0 10743747 121 irq266: em1:link 3893 0 cpu1:timer 5513791 62 cpu2:timer 5937620 66 cpu3:timer 84101501 948 Total 8904218168 100446 This server is using ipfw firewall with ipfw nat. The kernel is compiled with IPSEC support which has 8 rules in SPD table. The network card causing high load is em0 facing internal network. In ipfw there are just plain allow rules on it. All nat conversions and ipsec operations are done on another interface (em1) facing internet. I see both em0 and aac0 sharing irq 16 so maybe it is a RAID controller causing conflicts or whatever. Currently I can't find a solution or a method of more in-depth diagnostics of the matter. Please advice what can possibly cause the irq overload and how can I fight it? Kernel config: ========================================= cpu HAMMER ident SGP makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols # IPFW options IPFIREWALL #firewall options IPFIREWALL_VERBOSE #enable logging to syslogd(8) options IPFIREWALL_VERBOSE_LIMIT=300 #limit verbosity options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default options IPFIREWALL_FORWARD #packet destination changes options IPFIREWALL_NAT #ipfw kernel nat support options IPDIVERT #divert sockets # NAT options LIBALIAS # libalias library, performing NAT # IPSEC options IPSEC device crypto # Below is a common GENERIC config ========================================= dmesg output: ========================================= FreeBSD 9.1-RELEASE-p7 #2: Mon Sep 23 10:02:56 EEST 2013 ******@****************:/usr/obj/usr/src/sys/SGP amd64 CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (3392.37-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 Features=0xbfebfbff Features2=0x1f9ae3bf AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8141856768 (7764 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Event timer "HPET4" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 aac0: mem 0xf7800000-0xf79fffff irq 16 at device 0.0 on pci1 aac0: Enabling 64-bit address support aac0: Enable Raw I/O aac0: Enable 64-bit array aac0: New comm. interface enabled aac0: Adaptec 2405, aac driver 2.1.9-1 aacp0 on aac0 aacp1 on aac0 aacp2 on aac0 vgapci0: port 0xf000-0xf03f mem 0xf7400000-0xf77fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: aperture size is 256M, detected 65532k stolen memory pci0: at device 22.0 (no driver attached) ehci0: mem 0xf7e02000-0xf7e023ff irq 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 pcib2: irq 17 at device 28.0 on pci0 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 em0: port 0xe000-0xe03f mem 0xf7d40000-0xf7d5ffff,0xf7d20000-0xf7d3ffff irq 16 at device 0.0 on pci3 em0: Ethernet address: 90:e2:ba:17:59:d5 pcib4: irq 18 at device 28.2 on pci0 pci4: on pcib4 em1: port 0xd000-0xd01f mem 0xf7cc0000-0xf7cdffff,0xf7c00000-0xf7c7ffff,0xf7ce0000-0xf7ce3fff irq 18 at device 0.0 on pci4 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 68:05:ca:05:75:34 ehci1: mem 0xf7e01000-0xf7e013ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 isab0: at device 31.0 on pci0 isa0: on isab0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 orm0: at iomem 0xc0000-0xcdfff,0xce000-0xd47ff,0xd4800-0xd57ff,0xd5800-0xd67ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. ipfw2 (+ipv6) initialized, divert enabled, nat enabled, rule-based forwarding enabled, default to accept, logging disabled aacd0 on aac0 aacd0: 476150MB (975155200 sectors) usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: at usbus0 uhub2: on usbus0 ugen1.2: at usbus1 uhub3: on usbus1 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 ums0: on usbus0 ums0: 5 buttons and [XYZ] coordinates ID=1 pass0 at aacp0 bus 0 scbus0 target 0 lun 0 pass0: < ST3500630AS 3.AA> Fixed Uninstalled SCSI-5 device (offline) pass0: 3.300MB/s transfers pass1 at aacp0 bus 0 scbus0 target 1 lun 0 pass1: < ST3500630AS 3.AA> Fixed Uninstalled SCSI-5 device (offline) pass1: 3.300MB/s transfers SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! ========================================= # sysctl -a | grep dev.em dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.4 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x107c subvendor=0x8086 subdevice=0x1376 class=0x020000 dev.em.0.%parent: pci3 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 35079 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1077674561 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 47104 dev.em.0.fc_low_water: 45604 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 83 dev.em.0.txd_tail: 83 dev.em.0.rxd_head: 2 dev.em.0.rxd_tail: 1 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 15546581 dev.em.0.mac_stats.good_pkts_recvd: 15546581 dev.em.0.mac_stats.bcast_pkts_recvd: 74727 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 2240358 dev.em.0.mac_stats.rx_frames_65_127: 2026067 dev.em.0.mac_stats.rx_frames_128_255: 1667746 dev.em.0.mac_stats.rx_frames_256_511: 757367 dev.em.0.mac_stats.rx_frames_512_1023: 1081371 dev.em.0.mac_stats.rx_frames_1024_1522: 7773672 dev.em.0.mac_stats.good_octets_recvd: 11071669736 dev.em.0.mac_stats.good_octets_txd: 3680329427 dev.em.0.mac_stats.total_pkts_txd: 12769896 dev.em.0.mac_stats.good_pkts_txd: 12769896 dev.em.0.mac_stats.bcast_pkts_txd: 90 dev.em.0.mac_stats.mcast_pkts_txd: 2802292 dev.em.0.mac_stats.tx_frames_64: 5525466 dev.em.0.mac_stats.tx_frames_65_127: 2387324 dev.em.0.mac_stats.tx_frames_128_255: 2280501 dev.em.0.mac_stats.tx_frames_256_511: 337074 dev.em.0.mac_stats.tx_frames_512_1023: 204906 dev.em.0.mac_stats.tx_frames_1024_1522: 2034625 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.3.2 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=0 dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 dev.em.1.%parent: pci4 dev.em.1.nvm: -1 dev.em.1.debug: -1 dev.em.1.fc: 3 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.em.1.eee_control: 1 dev.em.1.link_irq: 3924 dev.em.1.mbuf_alloc_fail: 0 dev.em.1.cluster_alloc_fail: 0 dev.em.1.dropped: 0 dev.em.1.tx_dma_fail: 3 dev.em.1.rx_overruns: 0 dev.em.1.watchdog_timeouts: 0 dev.em.1.device_control: 1074790984 dev.em.1.rx_control: 67141634 dev.em.1.fc_high_water: 18432 dev.em.1.fc_low_water: 16932 dev.em.1.queue0.txd_head: 502 dev.em.1.queue0.txd_tail: 506 dev.em.1.queue0.tx_irq: 10924208 dev.em.1.queue0.no_desc_avail: 0 dev.em.1.queue0.rxd_head: 130 dev.em.1.queue0.rxd_tail: 129 dev.em.1.queue0.rx_irq: 12733678 dev.em.1.mac_stats.excess_coll: 142 dev.em.1.mac_stats.single_coll: 759301 dev.em.1.mac_stats.multiple_coll: 290042 dev.em.1.mac_stats.late_coll: 1049816 dev.em.1.mac_stats.collision_count: 1767409 dev.em.1.mac_stats.symbol_errors: 0 dev.em.1.mac_stats.sequence_errors: 0 dev.em.1.mac_stats.defer_count: 1959182 dev.em.1.mac_stats.missed_packets: 4794 dev.em.1.mac_stats.recv_no_buff: 0 dev.em.1.mac_stats.recv_undersize: 0 dev.em.1.mac_stats.recv_fragmented: 0 dev.em.1.mac_stats.recv_oversize: 0 dev.em.1.mac_stats.recv_jabber: 0 dev.em.1.mac_stats.recv_errs: 0 dev.em.1.mac_stats.crc_errs: 0 dev.em.1.mac_stats.alignment_errs: 0 dev.em.1.mac_stats.coll_ext_errs: 0 dev.em.1.mac_stats.xon_recvd: 0 dev.em.1.mac_stats.xon_txd: 0 dev.em.1.mac_stats.xoff_recvd: 0 dev.em.1.mac_stats.xoff_txd: 0 dev.em.1.mac_stats.total_pkts_recvd: 13461258 dev.em.1.mac_stats.good_pkts_recvd: 13456464 dev.em.1.mac_stats.bcast_pkts_recvd: 156733 dev.em.1.mac_stats.mcast_pkts_recvd: 0 dev.em.1.mac_stats.rx_frames_64: 1001246 dev.em.1.mac_stats.rx_frames_65_127: 5655384 dev.em.1.mac_stats.rx_frames_128_255: 3983776 dev.em.1.mac_stats.rx_frames_256_511: 539762 dev.em.1.mac_stats.rx_frames_512_1023: 260024 dev.em.1.mac_stats.rx_frames_1024_1522: 2016272 dev.em.1.mac_stats.good_octets_recvd: 4428966188 dev.em.1.mac_stats.good_octets_txd: 12327752567 dev.em.1.mac_stats.total_pkts_txd: 16190870 dev.em.1.mac_stats.good_pkts_txd: 16190870 dev.em.1.mac_stats.bcast_pkts_txd: 5 dev.em.1.mac_stats.mcast_pkts_txd: 0 dev.em.1.mac_stats.tx_frames_64: 547777 dev.em.1.mac_stats.tx_frames_65_127: 2630586 dev.em.1.mac_stats.tx_frames_128_255: 2404829 dev.em.1.mac_stats.tx_frames_256_511: 1283850 dev.em.1.mac_stats.tx_frames_512_1023: 1104051 dev.em.1.mac_stats.tx_frames_1024_1522: 8219777 dev.em.1.mac_stats.tso_txd: 234428 dev.em.1.mac_stats.tso_ctx_fail: 0 dev.em.1.interrupts.asserts: 1990 dev.em.1.interrupts.rx_pkt_timer: 0 dev.em.1.interrupts.rx_abs_timer: 0 dev.em.1.interrupts.tx_pkt_timer: 0 dev.em.1.interrupts.tx_abs_timer: 0 dev.em.1.interrupts.tx_queue_empty: 0 dev.em.1.interrupts.tx_queue_min_thresh: 0 dev.em.1.interrupts.rx_desc_min_thresh: 0 dev.em.1.interrupts.rx_overrun: 0 From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 08:47:29 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BA309A73 for ; Tue, 24 Sep 2013 08:47:29 +0000 (UTC) (envelope-from lists@rewt.org.uk) Received: from hosted.mx.as41113.net (abby.lhr1.as41113.net [91.208.177.20]) by mx1.freebsd.org (Postfix) with ESMTP id 2BE042BD6 for ; Tue, 24 Sep 2013 08:47:28 +0000 (UTC) Received: from [192.168.1.218] (unknown [212.9.98.193]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: lists@rewt.org.uk) by hosted.mx.as41113.net (Postfix) with ESMTPSA id 3ckbct5G6lzRS for ; Tue, 24 Sep 2013 09:47:26 +0100 (BST) Message-ID: <5241519C.9040908@rewt.org.uk> Date: Tue, 24 Sep 2013 09:47:24 +0100 From: Joe Holden User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Network stack changes References: <521E41CB.30700@yandex-team.ru> <523F4F14.9090404@yandex-team.ru> <201309240958.06172.zec@fer.hr> In-Reply-To: <201309240958.06172.zec@fer.hr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 08:47:29 -0000 On 24/09/2013 08:58, Marko Zec wrote: > On Tuesday 24 September 2013 00:46:46 Sami Halabi wrote: >> Hi, >> >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf>> et.unipi.it/~luigi/papers/20120601-dxr.pdf> >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff>> er.hr/dxr/stable_8_20120824.diff> >> >> I've tried the diff in 10-current, applied cleanly but had errors >> compiling new kernel... is there any work to make it work? i'd love to >> test it. > > Even if you'd make it compile on current, you could only run synthetic tests > measuring lookup performance using streams of random keys, as outlined in > the paper (btw. the paper at Luigi's site is an older draft, the final > version with slightly revised benchmarks is available here: > http://www.sigcomm.org/sites/default/files/ccr/papers/2012/October/2378956-2378961.pdf) > > I.e. the code only hooks into the routing API for testing purposes, but is > completely disconnected from the forwarding path. > aha! How much work would it be to enable it to be used? > We have a prototype in the works which combines DXR with Netmap in userspace > and is capable of sustaining well above line rate forwarding with > full-sized BGP views using Intel 10G cards on commodity multicore machines. > The work was somewhat stalled during the summer but I plan to wrap it up > and release the code until the end of this year. With recent advances in > netmap it might also be feasible to merge DXR and netmap entirely inside > the kernel but I've not explored that path yet... > mmm, forwarding using netmap would be pretty awesome... > Marko > > >> Sami >> >> >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < >> >> melifaro@yandex-team.ru> wrote: >>> On 29.08.2013 15:49, Adrian Chadd wrote: >>>> Hi, >>> >>> Hello Adrian! >>> I'm very sorry for the looong reply. >>> >>>> There's a lot of good stuff to review here, thanks! >>>> >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to >>>> keep locking things like that on a per-packet basis. We should be able >>>> to do this in a cleaner way - we can defer RX into a CPU pinned >>>> taskqueue and convert the interrupt handler to a fast handler that >>>> just schedules that taskqueue. We can ignore the ithread entirely >>>> here. >>>> >>>> What do you think? >>> >>> Well, it sounds good :) But performance numbers and Jack opinion is >>> more important :) >>> >>> Are you going to Malta? >>> >>>> Totally pie in the sky handwaving at this point: >>>> >>>> * create an array of mbuf pointers for completed mbufs; >>>> * populate the mbuf array; >>>> * pass the array up to ether_demux(). >>>> >>>> For vlan handling, it may end up populating its own list of mbufs to >>>> push up to ether_demux(). So maybe we should extend the API to have a >>>> bitmap of packets to actually handle from the array, so we can pass up >>>> a larger array of mbufs, note which ones are for the destination and >>>> then the upcall can mark which frames its consumed. >>>> >>>> I specifically wonder how much work/benefit we may see by doing: >>>> >>>> * batching packets into lists so various steps can batch process >>>> things rather than run to completion; >>>> * batching the processing of a list of frames under a single lock >>>> instance - eg, if the forwarding code could do the forwarding lookup >>>> for 'n' packets under a single lock, then pass that list of frames up >>>> to inet_pfil_hook() to do the work under one lock, etc, etc. >>> >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' due >>> to problem with egress interface pointer, as I mention earlier. However >>> it is interesting to see how much it helps, regardless of locking. >>> >>> Currently I'm thinking that we should try to change radix to something >>> different (it seems that it can be checked fast) and see what happened. >>> Luigi's performance numbers for our radix are too awful, and there is a >>> patch implementing alternative trie: >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf>> et.unipi.it/~luigi/papers/20120601-dxr.pdf> >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff>> er.hr/dxr/stable_8_20120824.diff> >>> >>>> Here, the processing would look less like "grab lock and process to >>>> completion" and more like "mark and sweep" - ie, we have a list of >>>> frames that we mark as needing processing and mark as having been >>>> processed at each layer, so we know where to next dispatch them. >>>> >>>> I still have some tool coding to do with PMC before I even think about >>>> tinkering with this as I'd like to measure stuff like per-packet >>>> latency as well as top-level processing overhead (ie, >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC >>>> interrupts on that core, etc.) >>> >>> That will be great to see! >>> >>>> Thanks, >>>> >>>> >>>> >>>> -adrian >>> >>> ______________________________**_________________ >>> freebsd-net@freebsd.org mailing list >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net>> eebsd.org/mailman/listinfo/freebsd-net> To unsubscribe, send any mail to >>> "freebsd-net-unsubscribe@**freebsd.org>> org> " >> >> -- >> Sami Halabi >> Information Systems Engineer >> NMS Projects Expert >> FreeBSD SysAdmin Expert >> >> >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < >> >> melifaro@yandex-team.ru> wrote: >>> On 29.08.2013 15:49, Adrian Chadd wrote: >>>> Hi, >>> >>> Hello Adrian! >>> I'm very sorry for the looong reply. >>> >>>> There's a lot of good stuff to review here, thanks! >>>> >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless to >>>> keep locking things like that on a per-packet basis. We should be able >>>> to do this in a cleaner way - we can defer RX into a CPU pinned >>>> taskqueue and convert the interrupt handler to a fast handler that >>>> just schedules that taskqueue. We can ignore the ithread entirely >>>> here. >>>> >>>> What do you think? >>> >>> Well, it sounds good :) But performance numbers and Jack opinion is >>> more important :) >>> >>> Are you going to Malta? >>> >>>> Totally pie in the sky handwaving at this point: >>>> >>>> * create an array of mbuf pointers for completed mbufs; >>>> * populate the mbuf array; >>>> * pass the array up to ether_demux(). >>>> >>>> For vlan handling, it may end up populating its own list of mbufs to >>>> push up to ether_demux(). So maybe we should extend the API to have a >>>> bitmap of packets to actually handle from the array, so we can pass up >>>> a larger array of mbufs, note which ones are for the destination and >>>> then the upcall can mark which frames its consumed. >>>> >>>> I specifically wonder how much work/benefit we may see by doing: >>>> >>>> * batching packets into lists so various steps can batch process >>>> things rather than run to completion; >>>> * batching the processing of a list of frames under a single lock >>>> instance - eg, if the forwarding code could do the forwarding lookup >>>> for 'n' packets under a single lock, then pass that list of frames up >>>> to inet_pfil_hook() to do the work under one lock, etc, etc. >>> >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' due >>> to problem with egress interface pointer, as I mention earlier. However >>> it is interesting to see how much it helps, regardless of locking. >>> >>> Currently I'm thinking that we should try to change radix to something >>> different (it seems that it can be checked fast) and see what happened. >>> Luigi's performance numbers for our radix are too awful, and there is a >>> patch implementing alternative trie: >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf>> et.unipi.it/~luigi/papers/20120601-dxr.pdf> >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff>> er.hr/dxr/stable_8_20120824.diff> >>> >>>> Here, the processing would look less like "grab lock and process to >>>> completion" and more like "mark and sweep" - ie, we have a list of >>>> frames that we mark as needing processing and mark as having been >>>> processed at each layer, so we know where to next dispatch them. >>>> >>>> I still have some tool coding to do with PMC before I even think about >>>> tinkering with this as I'd like to measure stuff like per-packet >>>> latency as well as top-level processing overhead (ie, >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC >>>> interrupts on that core, etc.) >>> >>> That will be great to see! >>> >>>> Thanks, >>>> >>>> >>>> >>>> -adrian >>> From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 09:41:36 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 33BCC1E2 for ; Tue, 24 Sep 2013 09:41:36 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.fer.hr (mail.fer.hr [161.53.72.233]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 533512E8B for ; Tue, 24 Sep 2013 09:41:34 +0000 (UTC) Received: from dyn10.nxlab.fer.hr (161.53.63.210) by MAIL.fer.hr (161.53.72.233) with Microsoft SMTP Server (TLS) id 14.2.342.3; Tue, 24 Sep 2013 11:41:32 +0200 From: Marko Zec To: Subject: Re: Network stack changes Date: Tue, 24 Sep 2013 11:42:24 +0200 User-Agent: KMail/1.9.10 References: <521E41CB.30700@yandex-team.ru> <201309240958.06172.zec@fer.hr> <5241519C.9040908@rewt.org.uk> In-Reply-To: <5241519C.9040908@rewt.org.uk> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201309241142.24542.zec@fer.hr> X-Originating-IP: [161.53.63.210] Cc: Joe Holden X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 09:41:36 -0000 On Tuesday 24 September 2013 10:47:24 Joe Holden wrote: > On 24/09/2013 08:58, Marko Zec wrote: > > On Tuesday 24 September 2013 00:46:46 Sami Halabi wrote: > >> Hi, > >> > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >>>.f er.hr/dxr/stable_8_20120824.diff> > >> > >> I've tried the diff in 10-current, applied cleanly but had errors > >> compiling new kernel... is there any work to make it work? i'd love to > >> test it. > > > > Even if you'd make it compile on current, you could only run synthetic > > tests measuring lookup performance using streams of random keys, as > > outlined in the paper (btw. the paper at Luigi's site is an older > > draft, the final version with slightly revised benchmarks is available > > here: > > http://www.sigcomm.org/sites/default/files/ccr/papers/2012/October/2378 > >956-2378961.pdf) > > > > I.e. the code only hooks into the routing API for testing purposes, but > > is completely disconnected from the forwarding path. > > aha! How much work would it be to enable it to be used? The inefficiency of our forwarding path is caused by so many factors combined (as others have put out previously in this thread) that simply plugging DXR into it wouldn't make a noticeable difference, except probably in scenarios with full BGP views loaded into the FIB, where our radix tree lookups can cause extreme cache trashing. Once we come closer to lockless operation from RX to TX queue, start processing packets and allocating / freeing buffers in batches, simplify mbufs or replace them entirely with a more efficient abstraction, then replacing radix tree lookups with DXR or something else might start making sense. Marko > > We have a prototype in the works which combines DXR with Netmap in > > userspace and is capable of sustaining well above line rate forwarding > > with full-sized BGP views using Intel 10G cards on commodity multicore > > machines. The work was somewhat stalled during the summer but I plan to > > wrap it up and release the code until the end of this year. With > > recent advances in netmap it might also be feasible to merge DXR and > > netmap entirely inside the kernel but I've not explored that path > > yet... > > mmm, forwarding using netmap would be pretty awesome... > > > Marko > > > >> Sami > >> > >> > >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > >> > >> melifaro@yandex-team.ru> wrote: > >>> On 29.08.2013 15:49, Adrian Chadd wrote: > >>>> Hi, > >>> > >>> Hello Adrian! > >>> I'm very sorry for the looong reply. > >>> > >>>> There's a lot of good stuff to review here, thanks! > >>>> > >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless > >>>> to keep locking things like that on a per-packet basis. We should be > >>>> able to do this in a cleaner way - we can defer RX into a CPU pinned > >>>> taskqueue and convert the interrupt handler to a fast handler that > >>>> just schedules that taskqueue. We can ignore the ithread entirely > >>>> here. > >>>> > >>>> What do you think? > >>> > >>> Well, it sounds good :) But performance numbers and Jack opinion is > >>> more important :) > >>> > >>> Are you going to Malta? > >>> > >>>> Totally pie in the sky handwaving at this point: > >>>> > >>>> * create an array of mbuf pointers for completed mbufs; > >>>> * populate the mbuf array; > >>>> * pass the array up to ether_demux(). > >>>> > >>>> For vlan handling, it may end up populating its own list of mbufs to > >>>> push up to ether_demux(). So maybe we should extend the API to have > >>>> a bitmap of packets to actually handle from the array, so we can > >>>> pass up a larger array of mbufs, note which ones are for the > >>>> destination and then the upcall can mark which frames its consumed. > >>>> > >>>> I specifically wonder how much work/benefit we may see by doing: > >>>> > >>>> * batching packets into lists so various steps can batch process > >>>> things rather than run to completion; > >>>> * batching the processing of a list of frames under a single lock > >>>> instance - eg, if the forwarding code could do the forwarding lookup > >>>> for 'n' packets under a single lock, then pass that list of frames > >>>> up to inet_pfil_hook() to do the work under one lock, etc, etc. > >>> > >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' > >>> due to problem with egress interface pointer, as I mention earlier. > >>> However it is interesting to see how much it helps, regardless of > >>> locking. > >>> > >>> Currently I'm thinking that we should try to change radix to > >>> something different (it seems that it can be checked fast) and see > >>> what happened. Luigi's performance numbers for our radix are too > >>> awful, and there is a patch implementing alternative trie: > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >>>.f er.hr/dxr/stable_8_20120824.diff> > >>> > >>>> Here, the processing would look less like "grab lock and process to > >>>> completion" and more like "mark and sweep" - ie, we have a list of > >>>> frames that we mark as needing processing and mark as having been > >>>> processed at each layer, so we know where to next dispatch them. > >>>> > >>>> I still have some tool coding to do with PMC before I even think > >>>> about tinkering with this as I'd like to measure stuff like > >>>> per-packet latency as well as top-level processing overhead (ie, > >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC > >>>> interrupts on that core, etc.) > >>> > >>> That will be great to see! > >>> > >>>> Thanks, > >>>> > >>>> > >>>> > >>>> -adrian > >>> > >>> ______________________________**_________________ > >>> freebsd-net@freebsd.org mailing list > >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net >>>fr eebsd.org/mailman/listinfo/freebsd-net> To unsubscribe, send any > >>> mail to > >>> "freebsd-net-unsubscribe@**freebsd.org >>>d. org> " > >> > >> -- > >> Sami Halabi > >> Information Systems Engineer > >> NMS Projects Expert > >> FreeBSD SysAdmin Expert > >> > >> > >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > >> > >> melifaro@yandex-team.ru> wrote: > >>> On 29.08.2013 15:49, Adrian Chadd wrote: > >>>> Hi, > >>> > >>> Hello Adrian! > >>> I'm very sorry for the looong reply. > >>> > >>>> There's a lot of good stuff to review here, thanks! > >>>> > >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointless > >>>> to keep locking things like that on a per-packet basis. We should be > >>>> able to do this in a cleaner way - we can defer RX into a CPU pinned > >>>> taskqueue and convert the interrupt handler to a fast handler that > >>>> just schedules that taskqueue. We can ignore the ithread entirely > >>>> here. > >>>> > >>>> What do you think? > >>> > >>> Well, it sounds good :) But performance numbers and Jack opinion is > >>> more important :) > >>> > >>> Are you going to Malta? > >>> > >>>> Totally pie in the sky handwaving at this point: > >>>> > >>>> * create an array of mbuf pointers for completed mbufs; > >>>> * populate the mbuf array; > >>>> * pass the array up to ether_demux(). > >>>> > >>>> For vlan handling, it may end up populating its own list of mbufs to > >>>> push up to ether_demux(). So maybe we should extend the API to have > >>>> a bitmap of packets to actually handle from the array, so we can > >>>> pass up a larger array of mbufs, note which ones are for the > >>>> destination and then the upcall can mark which frames its consumed. > >>>> > >>>> I specifically wonder how much work/benefit we may see by doing: > >>>> > >>>> * batching packets into lists so various steps can batch process > >>>> things rather than run to completion; > >>>> * batching the processing of a list of frames under a single lock > >>>> instance - eg, if the forwarding code could do the forwarding lookup > >>>> for 'n' packets under a single lock, then pass that list of frames > >>>> up to inet_pfil_hook() to do the work under one lock, etc, etc. > >>> > >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' > >>> due to problem with egress interface pointer, as I mention earlier. > >>> However it is interesting to see how much it helps, regardless of > >>> locking. > >>> > >>> Currently I'm thinking that we should try to change radix to > >>> something different (it seems that it can be checked fast) and see > >>> what happened. Luigi's performance numbers for our radix are too > >>> awful, and there is a patch implementing alternative trie: > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff >>>.f er.hr/dxr/stable_8_20120824.diff> > >>> > >>>> Here, the processing would look less like "grab lock and process to > >>>> completion" and more like "mark and sweep" - ie, we have a list of > >>>> frames that we mark as needing processing and mark as having been > >>>> processed at each layer, so we know where to next dispatch them. > >>>> > >>>> I still have some tool coding to do with PMC before I even think > >>>> about tinkering with this as I'd like to measure stuff like > >>>> per-packet latency as well as top-level processing overhead (ie, > >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NIC > >>>> interrupts on that core, etc.) > >>> > >>> That will be great to see! > >>> > >>>> Thanks, > >>>> > >>>> > >>>> > >>>> -adrian > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 10:27:58 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DF28C27A for ; Tue, 24 Sep 2013 10:27:58 +0000 (UTC) (envelope-from root@supermarket.cv.ua) Received: from supermarket.cv.ua (ec2-54-228-180-30.eu-west-1.compute.amazonaws.com [54.228.180.30]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5C67F2191 for ; Tue, 24 Sep 2013 10:27:58 +0000 (UTC) Received: from supermarket.cv.ua (localhost [127.0.0.1]) by supermarket.cv.ua (8.14.5/8.14.5) with ESMTP id r8OAJBk5019423; Tue, 24 Sep 2013 13:19:11 +0300 (EEST) (envelope-from root@supermarket.cv.ua) Received: from localhost (root@localhost) by supermarket.cv.ua (8.14.5/8.14.5/Submit) with ESMTP id r8OAJATm019420; Tue, 24 Sep 2013 13:19:10 +0300 (EEST) (envelope-from root@supermarket.cv.ua) Date: Tue, 24 Sep 2013 13:19:04 +0300 (EEST) From: Charlie Root To: Gabor Berczi Subject: Re: kern/182297: [cm] ArcNet driver fails to detect the link address - and does not work at all In-Reply-To: <201309231810.r8NIA01m064334@freefall.freebsd.org> Message-ID: References: <201309231810.r8NIA01m064334@freefall.freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 10:27:58 -0000 test message From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 16:55:12 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C55ED52C for ; Tue, 24 Sep 2013 16:55:12 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-ee0-x22f.google.com (mail-ee0-x22f.google.com [IPv6:2a00:1450:4013:c00::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 26B782A38 for ; Tue, 24 Sep 2013 16:55:12 +0000 (UTC) Received: by mail-ee0-f47.google.com with SMTP id d49so2585419eek.20 for ; Tue, 24 Sep 2013 09:55:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=+pKnm8ZlKMnTWZh7J7LNYiQvUrhEpI0ULkQGK2XnxCw=; b=vnjUAIyykoZy5LQ2uGc4Qw1msj3woTgtL2HrN0hoykG30nLM7k8+vPg3MhBoQc9qlR zUFrxgrZZJkMUpZkBlxTHl0hIJzNmHv3V5eOQNibIhHDJROkdQPo9GIQOzuoMSjvNnDW arMlgXTL4dSF2bRkxPJLxTd7IGQjw3m9NK3fOIe3N/wAW5IWjUMdJlqd+uD7joKPXxVB a+pUH+hKTE5+Uje+i0OCM8ImwXozbrjnay7M2MzL21aDhUpNybuM6a100tk577oj5yyP XSVvQT8M3Pp/oh33kzmsqV2kSZM++PvU4eHv8nlo6Y38WUjzR49ggzRa02t8hcbQExJS tAuA== MIME-Version: 1.0 X-Received: by 10.14.194.131 with SMTP id m3mr15912178een.45.1380041710414; Tue, 24 Sep 2013 09:55:10 -0700 (PDT) Received: by 10.223.152.72 with HTTP; Tue, 24 Sep 2013 09:55:10 -0700 (PDT) In-Reply-To: <201309241142.24542.zec@fer.hr> References: <521E41CB.30700@yandex-team.ru> <201309240958.06172.zec@fer.hr> <5241519C.9040908@rewt.org.uk> <201309241142.24542.zec@fer.hr> Date: Tue, 24 Sep 2013 09:55:10 -0700 Message-ID: Subject: Re: Network stack changes From: Vijay Singh To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 16:55:12 -0000 Hi, Robert Watson and Adrian have asked me to share some details around an MP design that we have at $work that is based on Robert's PCBGROUP work. Here are some details that I have put together. Design of the network MP system -=96------------------------------------------- 1. Reduce locking in the stack. 2. Should not require the use of specialized HW (NICs). 3. Should be able to leverage the capabilities of modern HW. The basic design is to introduce parallelism in the stack using the connection groups design implemented by Robert. We have a number of critical differences from the original design through. 1. The overall idea is not so much to provide CPU locality in our case, though the design doesn't necessarily preclude that. Since we run a complex workload, and have a run to completion model in the kernel, pinning threads causes unpredictable latencies. We find that a thread level CPU affinity implementation works quite well. 2. The number of connection groups created in the system are a constant, and as an optimization it is possible to set this number to the number of Rx/Tx queues in the NIC, but we don't do this. I will talk more about this later. 3. The number of threads created to service these queues is derived from the number of CPUs in the system. The thread library we used for our design is close to, but not quite the taskqueue(9). I can provide details if needed. 4. Input as well as output in this design for a given CG is processed by the thread currently assigned to the CG. Robert's design uses netisr but we found that it had 2 big issues. One, that it handles only mbufs as the objects that are queued. Second, and more importantly, it handles unidirectional traffic only. We felt that the original design was susceptible to locking at the connection level from the input & output threads contexts. 5. In order to avoid running too many threads, we use the fast interrupt handler, and enqueue the interrupt handler as a task for one of the threads created for the CG handling, as mentioned above. 6. We have changed the drivers to collect a fixed maximum number of mbufs, linked with m_nextpkt, and pass them up. I think this idea has been discussed previously on this list. The advantage we see is reduced locking in the Rx routine, as well as some possible optimizations using sw prefetching. 7. The driver also fills in the RSS hash computed by the HW in the mbuf. Code looks something like this: u16 pkt_info; pkt_info =3D (u16) (le16toh(cur->wb.lower.lo_dword.hs_rss.pkt_info) & IXGBE_RXDADV_RSSTYPE_MASK); if ((pkt_info =3D=3D IXGBE_RXDADV_RSSTYPE_IPV4_TCP)= || (pkt_info =3D=3D IXGBE_RXDADV_RSSTYPE_IPV6_TCP)= ) { sendmp->m_pkthdr.flowid =3D cur->wb.lower.hi_dword.rss; M_HASHTYPE_SET(sendmp, M_HASHTYPE_NIC); sendmp->m_flags |=3D M_FLOWID; } Here M_HASHTYPE_NIC is something we've added. 8. As packets flow up the stack, we've added an extension to the PCBGROUP design that allows a netisr module to provide a custom classifier (nh_m2flow) and a dispatch routine (nh_dispatcher). static struct netisr_handler ip_nh =3D { .nh_name =3D "ip", .nh_handler =3D ip_input, .nh_proto =3D NETISR_IP, .nh_policy =3D NETISR_POLICY_FLOW, .nh_dispatch =3D NETISR_DISPATCH_HYBRID, .nh_m2flow =3D bsd_cg_m2flow, .nh_dispatcher =3D bsd_cg_dispatcher, 9. In the m2flow routine we look at the mbuf. If the driver has supplied the RSS hash we use it, or else (for the loopback path for e.g.) we compute the RSS hash in sw. Something like.. if ((m !=3D NULL) && (m->m_flags & M_FLOWID) && (M_HASHTYPE_GET(m) =3D=3D M_HASHTYPE_NIC)) { if (is_v4) { M_HASHTYPE_SET(m, M_HASHTYPE_RSS_TCP_IPV4); ... else if (m) { th =3D (struct tcphdr *)(m->m_data + hlen); if (is_v4) { hash =3D bsd_cg_toeplitz_ipv4_hash(ip->ip_src.s_addr, ip->ip_dst.s_addr, th->th_sport, th->th_dport)= ; 10. In netisr_select_cpuid() we convert the flow id to CPU/CG ID using a simple modulo with the number of connection groups. Then we invoke the customer dispatcher to send the packets (with some batching) to the taskqueue like threads. The handler routine is the netisr handler of the protocol being processed, usually IP or IPv6. if (m->m_flags & M_FLOWID) { if (((dispatch_policy =3D=3D NETISR_DISPATCH_HYBRID= ) || (dispatch_policy =3D=3D NETISR_DISPATCH_DEFERR= ED)) && (npp->np_dispatcher !=3D NULL)) { *cpuidp =3D netisr_custom_flow2cpu(m->m_pkthdr.flowid); m =3D npp->np_dispatcher(m, *cpuidp, proto)= ; if (m =3D=3D NULL) return (NULL); } } 11. The hash placed in the mbuf is then used to lookup the pcb in the corresponding pcbgroup table: static __inline u_int in_pcbgroup_getbucket(struct inpcbinfo *pcbinfo, uint32_t hash) { return (hash % pcbinfo->ipi_npcbgroups); } /* * Map a (hashtype, hash) tuple into a connection group, or NULL if the has= h * information is insufficient to identify the pcbgroup. */ struct inpcbgroup * in_pcbgroup_byhash(struct inpcbinfo *pcbinfo, u_int hashtype, uint32_t hash= ) { if ((pcbinfo->ipi_hashfields =3D=3D IPI_HASHFIELDS_4TUPLE && hashtype =3D=3D M_HASHTYPE_RSS_TCP_IPV4) || (pcbinfo->ipi_hashfields =3D=3D IPI_HASHFIELDS_2TUPLE && hashtype =3D=3D M_HASHTYPE_RSS_IPV4)) { return (&pcbinfo->ipi_pcbgroups[ in_pcbgroup_getbucket(pcbinfo, hash)]); } return (NULL); } I wanted to pass on this information as a starting point. I'll be at the vendor summit in Nov. I think it has been suggested that network performance be one of the discussion points. I'd be happy to participate and provide inputs based on our own experience. -vijay On Tue, Sep 24, 2013 at 2:42 AM, Marko Zec wrote: > On Tuesday 24 September 2013 10:47:24 Joe Holden wrote: > > On 24/09/2013 08:58, Marko Zec wrote: > > > On Tuesday 24 September 2013 00:46:46 Sami Halabi wrote: > > >> Hi, > > >> > > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf< > http://info > > >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff< > http://www.nxlab > > >>>.f er.hr/dxr/stable_8_20120824.diff> > > >> > > >> I've tried the diff in 10-current, applied cleanly but had errors > > >> compiling new kernel... is there any work to make it work? i'd love = to > > >> test it. > > > > > > Even if you'd make it compile on current, you could only run syntheti= c > > > tests measuring lookup performance using streams of random keys, as > > > outlined in the paper (btw. the paper at Luigi's site is an older > > > draft, the final version with slightly revised benchmarks is availabl= e > > > here: > > > > http://www.sigcomm.org/sites/default/files/ccr/papers/2012/October/2378 > > >956-2378961.pdf) > > > > > > I.e. the code only hooks into the routing API for testing purposes, b= ut > > > is completely disconnected from the forwarding path. > > > > aha! How much work would it be to enable it to be used? > > The inefficiency of our forwarding path is caused by so many factors > combined (as others have put out previously in this thread) that simply > plugging DXR into it wouldn't make a noticeable difference, except probab= ly > in scenarios with full BGP views loaded into the FIB, where our radix tre= e > lookups can cause extreme cache trashing. > > Once we come closer to lockless operation from RX to TX queue, start > processing packets and allocating / freeing buffers in batches, simplify > mbufs or replace them entirely with a more efficient abstraction, then > replacing radix tree lookups with DXR or something else might start makin= g > sense. > > Marko > > > > We have a prototype in the works which combines DXR with Netmap in > > > userspace and is capable of sustaining well above line rate forwardin= g > > > with full-sized BGP views using Intel 10G cards on commodity multicor= e > > > machines. The work was somewhat stalled during the summer but I plan = to > > > wrap it up and release the code until the end of this year. With > > > recent advances in netmap it might also be feasible to merge DXR and > > > netmap entirely inside the kernel but I've not explored that path > > > yet... > > > > mmm, forwarding using netmap would be pretty awesome... > > > > > Marko > > > > > >> Sami > > >> > > >> > > >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > > >> > > >> melifaro@yandex-team.ru> wrote: > > >>> On 29.08.2013 15:49, Adrian Chadd wrote: > > >>>> Hi, > > >>> > > >>> Hello Adrian! > > >>> I'm very sorry for the looong reply. > > >>> > > >>>> There's a lot of good stuff to review here, thanks! > > >>>> > > >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointles= s > > >>>> to keep locking things like that on a per-packet basis. We should = be > > >>>> able to do this in a cleaner way - we can defer RX into a CPU pinn= ed > > >>>> taskqueue and convert the interrupt handler to a fast handler that > > >>>> just schedules that taskqueue. We can ignore the ithread entirely > > >>>> here. > > >>>> > > >>>> What do you think? > > >>> > > >>> Well, it sounds good :) But performance numbers and Jack opinion is > > >>> more important :) > > >>> > > >>> Are you going to Malta? > > >>> > > >>>> Totally pie in the sky handwaving at this point: > > >>>> > > >>>> * create an array of mbuf pointers for completed mbufs; > > >>>> * populate the mbuf array; > > >>>> * pass the array up to ether_demux(). > > >>>> > > >>>> For vlan handling, it may end up populating its own list of mbufs = to > > >>>> push up to ether_demux(). So maybe we should extend the API to hav= e > > >>>> a bitmap of packets to actually handle from the array, so we can > > >>>> pass up a larger array of mbufs, note which ones are for the > > >>>> destination and then the upcall can mark which frames its consumed= . > > >>>> > > >>>> I specifically wonder how much work/benefit we may see by doing: > > >>>> > > >>>> * batching packets into lists so various steps can batch process > > >>>> things rather than run to completion; > > >>>> * batching the processing of a list of frames under a single lock > > >>>> instance - eg, if the forwarding code could do the forwarding look= up > > >>>> for 'n' packets under a single lock, then pass that list of frames > > >>>> up to inet_pfil_hook() to do the work under one lock, etc, etc. > > >>> > > >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' > > >>> due to problem with egress interface pointer, as I mention earlier. > > >>> However it is interesting to see how much it helps, regardless of > > >>> locking. > > >>> > > >>> Currently I'm thinking that we should try to change radix to > > >>> something different (it seems that it can be checked fast) and see > > >>> what happened. Luigi's performance numbers for our radix are too > > >>> awful, and there is a patch implementing alternative trie: > > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf< > http://info > > >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff< > http://www.nxlab > > >>>.f er.hr/dxr/stable_8_20120824.diff> > > >>> > > >>>> Here, the processing would look less like "grab lock and process t= o > > >>>> completion" and more like "mark and sweep" - ie, we have a list of > > >>>> frames that we mark as needing processing and mark as having been > > >>>> processed at each layer, so we know where to next dispatch them. > > >>>> > > >>>> I still have some tool coding to do with PMC before I even think > > >>>> about tinkering with this as I'd like to measure stuff like > > >>>> per-packet latency as well as top-level processing overhead (ie, > > >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NI= C > > >>>> interrupts on that core, etc.) > > >>> > > >>> That will be great to see! > > >>> > > >>>> Thanks, > > >>>> > > >>>> > > >>>> > > >>>> -adrian > > >>> > > >>> ______________________________**_________________ > > >>> freebsd-net@freebsd.org mailing list > > >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-net . > > >>>fr eebsd.org/mailman/listinfo/freebsd-net> To unsubscribe, send any > > >>> mail to > > >>> "freebsd-net-unsubscribe@**freebsd.org > > >>>d. org> " > > >> > > >> -- > > >> Sami Halabi > > >> Information Systems Engineer > > >> NMS Projects Expert > > >> FreeBSD SysAdmin Expert > > >> > > >> > > >> On Sun, Sep 22, 2013 at 11:12 PM, Alexander V. Chernikov < > > >> > > >> melifaro@yandex-team.ru> wrote: > > >>> On 29.08.2013 15:49, Adrian Chadd wrote: > > >>>> Hi, > > >>> > > >>> Hello Adrian! > > >>> I'm very sorry for the looong reply. > > >>> > > >>>> There's a lot of good stuff to review here, thanks! > > >>>> > > >>>> Yes, the ixgbe RX lock needs to die in a fire. It's kinda pointles= s > > >>>> to keep locking things like that on a per-packet basis. We should = be > > >>>> able to do this in a cleaner way - we can defer RX into a CPU pinn= ed > > >>>> taskqueue and convert the interrupt handler to a fast handler that > > >>>> just schedules that taskqueue. We can ignore the ithread entirely > > >>>> here. > > >>>> > > >>>> What do you think? > > >>> > > >>> Well, it sounds good :) But performance numbers and Jack opinion is > > >>> more important :) > > >>> > > >>> Are you going to Malta? > > >>> > > >>>> Totally pie in the sky handwaving at this point: > > >>>> > > >>>> * create an array of mbuf pointers for completed mbufs; > > >>>> * populate the mbuf array; > > >>>> * pass the array up to ether_demux(). > > >>>> > > >>>> For vlan handling, it may end up populating its own list of mbufs = to > > >>>> push up to ether_demux(). So maybe we should extend the API to hav= e > > >>>> a bitmap of packets to actually handle from the array, so we can > > >>>> pass up a larger array of mbufs, note which ones are for the > > >>>> destination and then the upcall can mark which frames its consumed= . > > >>>> > > >>>> I specifically wonder how much work/benefit we may see by doing: > > >>>> > > >>>> * batching packets into lists so various steps can batch process > > >>>> things rather than run to completion; > > >>>> * batching the processing of a list of frames under a single lock > > >>>> instance - eg, if the forwarding code could do the forwarding look= up > > >>>> for 'n' packets under a single lock, then pass that list of frames > > >>>> up to inet_pfil_hook() to do the work under one lock, etc, etc. > > >>> > > >>> I'm thinking the same way, but we're stuck with 'forwarding lookup' > > >>> due to problem with egress interface pointer, as I mention earlier. > > >>> However it is interesting to see how much it helps, regardless of > > >>> locking. > > >>> > > >>> Currently I'm thinking that we should try to change radix to > > >>> something different (it seems that it can be checked fast) and see > > >>> what happened. Luigi's performance numbers for our radix are too > > >>> awful, and there is a patch implementing alternative trie: > > >>> http://info.iet.unipi.it/~**luigi/papers/20120601-dxr.pdf< > http://info > > >>>.i et.unipi.it/~luigi/papers/20120601-dxr.pdf> > > >>> http://www.nxlab.fer.hr/dxr/**stable_8_20120824.diff< > http://www.nxlab > > >>>.f er.hr/dxr/stable_8_20120824.diff> > > >>> > > >>>> Here, the processing would look less like "grab lock and process t= o > > >>>> completion" and more like "mark and sweep" - ie, we have a list of > > >>>> frames that we mark as needing processing and mark as having been > > >>>> processed at each layer, so we know where to next dispatch them. > > >>>> > > >>>> I still have some tool coding to do with PMC before I even think > > >>>> about tinkering with this as I'd like to measure stuff like > > >>>> per-packet latency as well as top-level processing overhead (ie, > > >>>> CPU_CLK_UNHALTED.THREAD_P / lagg0 TX bytes/pkts, RX bytes/pkts, NI= C > > >>>> interrupts on that core, etc.) > > >>> > > >>> That will be great to see! > > >>> > > >>>> Thanks, > > >>>> > > >>>> > > >>>> > > >>>> -adrian > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 17:56:03 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 980BDF3C for ; Tue, 24 Sep 2013 17:56:03 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from rush.bluerosetech.com (rush.bluerosetech.com [IPv6:2607:fc50:1000:9b00::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6F3F92E49 for ; Tue, 24 Sep 2013 17:56:03 +0000 (UTC) Received: from chombo.houseloki.net (c-76-27-220-79.hsd1.wa.comcast.net [76.27.220.79]) by rush.bluerosetech.com (Postfix) with ESMTPSA id 77BC811434; Tue, 24 Sep 2013 10:55:54 -0700 (PDT) Received: from [IPv6:2601:7:1680:365:dad:bc45:f3db:a353] (unknown [IPv6:2601:7:1680:365:dad:bc45:f3db:a353]) by chombo.houseloki.net (Postfix) with ESMTPSA id 961313BD; Tue, 24 Sep 2013 10:55:52 -0700 (PDT) Message-ID: <5241D225.8090507@bluerosetech.com> Date: Tue, 24 Sep 2013 10:55:49 -0700 From: Darren Pilgrim User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: Yar Tikhiy , freebsd-net@freebsd.org Subject: Re: Free book draft: IPv6 for IPv4 Experts References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 17:56:03 -0000 On 9/23/2013 4:50 AM, Yar Tikhiy wrote: > The project page is: https://sites.google.com/site/yartikhiy/home/ipv6book Do you have a crowd-funding campaign for this? From owner-freebsd-net@FreeBSD.ORG Tue Sep 24 18:58:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:1900:2254:206a::19:2]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 18BC61F2 for ; Tue, 24 Sep 2013 18:58:19 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from butcher-nb.yandex.net (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) by mx2.freebsd.org (Postfix) with ESMTP id 9C366C77 for ; Tue, 24 Sep 2013 18:58:18 +0000 (UTC) Message-ID: <5241DFC9.6070802@FreeBSD.org> Date: Tue, 24 Sep 2013 22:54:01 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: [RFC] Don't embed scope zone id into IPv6 addresses X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2013 18:58:19 -0000 Hi All, As many of you know, FreeBSD uses IPv6 stack derived from the KAME project. The KAME project was started so long ago, when many RFCs and standards weren't written yet. Also the struct sockaddr_in6 did not have the sin6_scope_id field and link-local addresses had to be disambiguated in an implementation-dependent manner. The KAME-specific workaround to this problem was use of an embedded form. I.e. in the kernel we keep scope zone id in the second 16-bit field of IPv6 address structure. Also we have a bunch of helper functions in the kernel to maintain embedded form of addresses: in6_setscope(), in6_clearscope(), sa6_recoverscope(), sa6_embedscope(). Conversion to embedded form and back can be done several times per packet [1]. This is basically useless operations. And it is worth mentioning that some operations with embedded form use locking, and also changing an address on packet flow has bad effect for caching. My proposal is remove this code from our kernel. From my investigation this needs the following changes: 1. Do not keep link-local addresses in the routing tables. We already have list and hash table with all IPv6 addresses, so we can use sin6_scope_id to disambiguate an address and determine outgoing interface. 2. Add scope zone id field to the struct in_endpoints and hc_metrics. This also leads to changes in the PCB code. 3. Rework ip6_input * Don't use auxiliary data * Use ifaddr hash to check against own addresses * Remove route and lle lookups from input path 4. Rework ip6_output * Introduce new IPV6_USEROIF flag and use ifpp argument to specify outgoing interface * Rework route lookup part of ip6_output() to properly handle the fact, that now we don't keep routes for link-local addresses. It is the biggest changes, also there are many of small changes that should be done. I would like to see your opinions about first point. [1] http://tinyurl.com/figure2-12 -- WBR, Andrey V. Elsukov From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 03:32:20 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E43EAB68 for ; Wed, 25 Sep 2013 03:32:20 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x232.google.com (mail-ee0-x232.google.com [IPv6:2a00:1450:4013:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6AAC0228C for ; Wed, 25 Sep 2013 03:32:20 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id d51so2834763eek.23 for ; Tue, 24 Sep 2013 20:32:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=QZ0JEsCe+NRMEOYB8MO3pMvUpcoYKZUd6n3Gm4vaEec=; b=fK9fPJfQwc+gzg1q8OHBIEeA/xH3PjOw7jY9+9pnp5n5mPqqeHmYx9PHwXnkrSEnhF HOcgo+gMQAwe/KVpIoi7LL1JeyuBYxkVSO+Et2IWleGmfj5GepiW/VGlXhsQCoKdUyRe dmCFGIVmRbXxt9AH0xsmORG208dtXmZXea9KoaeJ6dTVO7XEcnl4emPjRplXOtdSI5f9 OizlrL3vxRpgiBAa/o3XTArNfa3Bk5+JdWN7MsXK+FyT9DnyiAVTT4w6KmpDA76bsb98 zFIKpjoPjLH/stzYPc3aR4SSWvxyL+OKL/2zxIlr6lIFtpmojN2EKP30mFQuZpDMnx4y 4RUQ== MIME-Version: 1.0 X-Received: by 10.14.181.194 with SMTP id l42mr62719eem.63.1380079938619; Tue, 24 Sep 2013 20:32:18 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Tue, 24 Sep 2013 20:32:18 -0700 (PDT) Date: Tue, 24 Sep 2013 20:32:18 -0700 Message-ID: Subject: netmap: understanding pkg-gen.c From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 03:32:21 -0000 I am new to netmap so thought of confirming my understanding. I was trying to associate ixgbe interface with netmap and blast pkts to it using tools/tools/netmap/pkg-gen.c My setup looks like this: FreeBSD 10.0-ALPHA1 #2: Sun Sep 22 18:08:18 UTC 2013 -bash-4.2$ sysctl hw.physmem hw.physmem: 51505954816 -bash-4.2$ sysctl hw.ncpu hw.ncpu: 24 -bash-4.2$ sysctl hw.ix hw.ix.enable_aim: 1 hw.ix.max_interrupt_rate: 31250 hw.ix.rx_process_limit: 256 hw.ix.tx_process_limit: 256 hw.ix.enable_msix: 1 hw.ix.num_queues: 8 hw.ix.txd: 2048 hw.ix.rxd: 2048 -bash-4.2$ On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 (10.73.149.17) and this is how I am using this binary: -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s 10.73.149.28 extract_ip_range [143] extract IP range from 10.73.149.28 extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0 extract_ip_range [143] extract IP range from 10.73.149.17 extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0 extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5 extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5 extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff main [1530] map size is 334980 Kb main [1552] mapping 334980 Kbytes Sending on ix1: 8 queues, 8 threads and 8 cpus. 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff) main [1622] Sending 512 packets every 0.000000000 ns main [1624] Wait 2 secs for phy reset main [1626] Ready... sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [775] start sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy sender_body [848] drop copy main_thread [1192] 15024157 pps (15050104 pkts in 1001727 usec) main_thread [1192] 14882290 pps (14900223 pkts in 1001205 usec) main_thread [1192] 14879515 pps (14903798 pkts in 1001632 usec) main_thread [1192] 14880924 pps (15795952 pkts in 1061490 usec) main_thread [1192] 14881411 pps (15821633 pkts in 1063181 usec) main_thread [1192] 14880095 pps (15427549 pkts in 1036791 usec) main_thread [1192] 7986707 pps (8100741 pkts in 1014278 usec) Sent 100000000 packets, 60 bytes each, in 6.71 seconds. Speed: 14.90 Mpps Bandwidth: 7.15 Gbps (raw 10.01 Gbps) $ top -H shows: 25853 root 24 0 348M 35692K select 16 0:01 7.28% pkt-gen{pkt-gen} 25853 root 25 0 348M 35692K select 21 0:01 7.28% pkt-gen{pkt-gen} 25853 root 25 0 348M 35692K select 15 0:01 7.28% pkt-gen{pkt-gen} 25853 root 23 0 348M 35692K select 12 0:01 6.40% pkt-gen{pkt-gen} 25853 root 24 0 348M 35692K select 22 0:01 6.30% pkt-gen{pkt-gen} 25853 root 23 0 348M 35692K select 9 0:01 5.66% pkt-gen{pkt-gen} 25853 root 23 0 348M 35692K select 7 0:01 5.57% pkt-gen{pkt-gen} 25853 root 23 0 348M 35692K CPU0 0 0:01 5.27% pkt-gen{pkt-gen} 12 root -92 - 0K 1184K WAIT 6 0:10 5.08% intr{irq290: ix1:que } 12 root -92 - 0K 1184K WAIT 7 0:09 4.98% intr{irq291: ix1:que } 12 root -92 - 0K 1184K WAIT 5 0:08 4.79% intr{irq289: ix1:que } 12 root -92 - 0K 1184K WAIT 2 0:15 4.69% intr{irq286: ix1:que } 12 root -92 - 0K 1184K WAIT 3 0:08 4.69% intr{irq287: ix1:que } 12 root -92 - 0K 1184K WAIT 4 0:14 4.59% intr{irq288: ix1:que } 12 root -92 - 0K 1184K WAIT 1 0:06 3.96% intr{irq285: ix1:que } 12 root -92 - 0K 1184K WAIT 0 0:05 0.98% intr{irq284: ix1:que } I can only specify -p (threads) upto 8 because it cannot be more than the hw.ix.num_queues=8, is that correct? Cheers, Hiren From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 03:37:51 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4B6A1C5A for ; Wed, 25 Sep 2013 03:37:51 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CDA9722D0 for ; Wed, 25 Sep 2013 03:37:50 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id gx14so4445769lab.37 for ; Tue, 24 Sep 2013 20:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=1zA+yw6vF5hz6jx3chwyy9bv/SWAN93p/8dU8Z9ORbs=; b=HQbud9u9qo+bLzWsex9mj8n9M7Oi44eSEQbICwSB6OIr/h6NEaoJea6GhqnZ01Y0K6 dqyd2RgBHnb5gDOGa5J+jQM2JVNYPl+d9k9s4O50f0wRr4eVn1O2zrPkfGTvUaLmDxbj EcWozlJH+aQc9x0fniTXjV6QmkfOe7Ltucv7GX2wjvr4w9UIWREDzgOWeX7HsuOikfiL tqUVG8ZmRZGdYHi832klEJyODcznh2X3MC4/jANlUT5uNVCM01lJJlwljIgqIMztmOYP UnMqTDVBq52lrh58HcR204dUxzm59mkeNzgudlbUI8hbub+QgNoJ5Gi4gOWK2LrPy9ql VIdg== MIME-Version: 1.0 X-Received: by 10.112.57.49 with SMTP id f17mr9930073lbq.26.1380080268703; Tue, 24 Sep 2013 20:37:48 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.105 with HTTP; Tue, 24 Sep 2013 20:37:48 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 05:37:48 +0200 X-Google-Sender-Auth: bhGKh6NHp7zh1pY0ulPiqR7wQ_I Message-ID: Subject: Re: netmap: understanding pkg-gen.c From: Luigi Rizzo To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 03:37:51 -0000 On Wed, Sep 25, 2013 at 5:32 AM, hiren panchasara wrote: > I am new to netmap so thought of confirming my understanding. > ... > > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1 (10.73.149.17) > and this is how I am using this binary: > > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d > 10.73.149.17 -s 10.73.149.28 ... > I can only specify -p (threads) upto 8 because it cannot be more than the > hw.ix.num_queues=8, is that correct? correct. it's either 1 or the total number of queues. And you can almost surely get line rate with a single thread. cheers luigi From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 06:54:01 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0243B428 for ; Wed, 25 Sep 2013 06:54:01 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x233.google.com (mail-ee0-x233.google.com [IPv6:2a00:1450:4013:c00::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6A0372C0A for ; Wed, 25 Sep 2013 06:54:00 +0000 (UTC) Received: by mail-ee0-f51.google.com with SMTP id c1so2926134eek.38 for ; Tue, 24 Sep 2013 23:53:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=4OfoQxQYe5OCbYNIolZ8dZLz7eP9AcIc+RAawbkr8+g=; b=Pxsdtq21LdtWLPxajZp/ZsvoOeUTRCGFRG9MEa6cXMyp/62WOCz8+tRvG4z2T/jgoI qnlIipWgg+6jpibPLcVsATvDx8xiRP4N2slw6chELTqNktzZVsE6hiCGcjcd5dz9ujAe KCeI/JsqLMZ9goAQrYJPOnAhAw+0s8TgBIhUzDQcFu4ay4Uz7u5wMWdYgjTqf3N4nPKq nyMAQuBTtRWtd5o+dPRKvcAo3ly/PaLVHlNl9xO8P5tM9EYu/gHIghGB1AdL1grk/Ck4 YofvE8ECdPzBRtUXTi2sg9I1oqqzM7Vfo5hY6O8Z3NX8cmd8IUpGmSpl3U+kRBE00602 O/zg== MIME-Version: 1.0 X-Received: by 10.14.104.199 with SMTP id i47mr322521eeg.85.1380092038644; Tue, 24 Sep 2013 23:53:58 -0700 (PDT) Sender: hiren.panchasara@gmail.com Received: by 10.14.105.137 with HTTP; Tue, 24 Sep 2013 23:53:58 -0700 (PDT) In-Reply-To: References: Date: Tue, 24 Sep 2013 23:53:58 -0700 X-Google-Sender-Auth: INuQlZiAtLeZNTQZHrqzco78AaU Message-ID: Subject: Re: Exposing sysctls for ixgbe From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 06:54:01 -0000 Any comments? cheers, Hiren On Sun, Sep 22, 2013 at 12:01 PM, hiren panchasara wrote: > $ sysctl hw.igb > hw.igb.rxd: 4096 > hw.igb.txd: 4096 > hw.igb.enable_aim: 1 > hw.igb.enable_msix: 1 > hw.igb.max_interrupt_rate: 8000 > hw.igb.buf_ring_size: 4096 > hw.igb.header_split: 0 > hw.igb.num_queues: 1 > hw.igb.rx_process_limit: 100 > $ sysctl hw.ix > sysctl: unknown oid 'hw.ix': No such file or directory > > I thought it would be nice to have these things exposed. So I copied them > from igb: > http://people.freebsd.org/~hiren/ixgbe_sysctls.txt > > Changes for if_igb.c is to expose correct auto-tuned value for a running > system for "hw.igb.num_queues", which is not the case right now. > > Thanks to markj@ for help/pointers. > > Please let me know if the diffs look okay. > > cheers, > Hiren > From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 08:07:27 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B23F264E for ; Wed, 25 Sep 2013 08:07:27 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x229.google.com (mail-ee0-x229.google.com [IPv6:2a00:1450:4013:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4EFC52062 for ; Wed, 25 Sep 2013 08:07:27 +0000 (UTC) Received: by mail-ee0-f41.google.com with SMTP id d17so3018739eek.28 for ; Wed, 25 Sep 2013 01:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=efURvtG1oU30jYpdG13v8gyNw2d3Df3zPnH0lErBZBg=; b=xVXDxTd2d+J5myUs0mpCbUrZ+nYwEasiE1R9Q7RQY7TgsMMGvUof6VTnlsZntbPcM0 Ny1X7Y4mNnWsowEnV3KTZcoM2uWUrE2CEJJOYucjm0LtFKSibcaf9wSYeKdO+kK4IbsE l1hP5A1Si6b1UlAcf+R4H+FLb8dvRngrMblHaTLwRHEMfpByRObD4GKHsBPlvYry01ml HlxNMMsJSodcaQhS0PoF3SdxgN1wKUhvlbmqhWfkPKNcYoZ8xu0794BBhh0tytLfJvc/ FB4udCYyCJO3hWdbblQZaDk9kRnRod9dY8JYGnJsvza/oBRsyLOJ5Hy74f1dcYFW4Dwe ZIbw== MIME-Version: 1.0 X-Received: by 10.14.198.197 with SMTP id v45mr2732824een.52.1380096445701; Wed, 25 Sep 2013 01:07:25 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 25 Sep 2013 01:07:25 -0700 (PDT) Date: Wed, 25 Sep 2013 01:07:25 -0700 Message-ID: Subject: netmap: traffic distribution From: hiren panchasara To: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 08:07:27 -0000 I am providing line rate traffic (via pkg-gen.c) to my 10gig ix interface. Now on receiving side, is there a way to sub-divide the traffic into multiple workloads using netmap? For example, can I get two 5G flows from 10Gbps traffic? cheers, Hiren From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 08:22:43 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 01CBCA51 for ; Wed, 25 Sep 2013 08:22:42 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 820D3217D for ; Wed, 25 Sep 2013 08:22:42 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id y6so4705810lbh.21 for ; Wed, 25 Sep 2013 01:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/nMYwMHsbl/UQ7b8IXfsgIjmHZoMdIiKZw45o8yAAYM=; b=Mx+uHQOR6VeaVuVR6TWaE9cMx5RDkYDV8j5LkueuCT1EfUowR264f+n2scF6qKT2n2 ngbSVtc3lnclZqDaSqcC2cqpLilW9sTkjHD/cLxMhF/7/7g1gWIKZqcebvlImvH9geZe 9a/wT7JyZVdVMu7sRql7Pa9eFHBwvEUDRfHoWfpTgDGi5ke2fNq4IWxeQJJIyCrYr4BL ZJmw9pSWQXWbT4QIAJwgK59ly/7BxsGr7u3+84/Oh3QoiU5i9oYINv8wy1PG84H6ZvSB Y52NMnczf8otuyqgF4msV0sP8lOoUhS4g/R+iLO4uMD4WqiRzcwHFY7Du2/UW7BVUNnV Z7WA== MIME-Version: 1.0 X-Received: by 10.152.7.100 with SMTP id i4mr674506laa.35.1380097360370; Wed, 25 Sep 2013 01:22:40 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.105 with HTTP; Wed, 25 Sep 2013 01:22:40 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 10:22:40 +0200 X-Google-Sender-Auth: MDKOlErP92MI26kUjxlM8oZT01g Message-ID: Subject: Re: netmap: traffic distribution From: Luigi Rizzo To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 08:22:43 -0000 On Wed, Sep 25, 2013 at 10:07 AM, hiren panchasara wrote: > > I am providing line rate traffic (via pkg-gen.c) to my 10gig ix interface. > > Now on receiving side, is there a way to sub-divide the traffic into > multiple workloads using netmap? > > For example, can I get two 5G flows from 10Gbps traffic? not directly. You'd need to send packets with different addresses that match the way the filters on the NIC (RSS or similar) are programmed. cheers luigi From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 08:54:01 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7020E12C for ; Wed, 25 Sep 2013 08:54:01 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x233.google.com (mail-ee0-x233.google.com [IPv6:2a00:1450:4013:c00::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 081562359 for ; Wed, 25 Sep 2013 08:54:00 +0000 (UTC) Received: by mail-ee0-f51.google.com with SMTP id c1so2982437eek.24 for ; Wed, 25 Sep 2013 01:53:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vSTSyTlmUH6F0JAthoYGzO1b/irT0oPCA3GLbr6zjbs=; b=NgsYcHCRjDBMTQZRIEfCxRMJoYJEzS6rpxqsiucqG21zo14wUxqHX0t2EK/611MtO+ BOquxc4Rqg67j0ZC4/Tmk9yYOeDfXgNGQyQXptwopVPi5jzR/MXE9bLrmpgD9YieSsmh 927bCJ6WGwUdQ7uYvBIzFh+1EG90TPiXYuoRYN7OulIJQuy6QhIn2qETDRmjDm9OMZ0G BPSPKjT1fy2yL4xSL1RmT4FW+PnW+XuMRKBkp27LsO2LaYxJ55PIkYb7AvAOcDLtpWnR gET+MiESm8bHtRy/NMV6OlU+UBWv6KOAY4PCxFpW55cTg0+uu26kiIo0PMtgJAVpOZZd M6rQ== MIME-Version: 1.0 X-Received: by 10.15.74.197 with SMTP id j45mr29414377eey.40.1380099239479; Wed, 25 Sep 2013 01:53:59 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 25 Sep 2013 01:53:59 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 01:53:59 -0700 Message-ID: Subject: Re: netmap: traffic distribution From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 08:54:01 -0000 On Wed, Sep 25, 2013 at 1:22 AM, Luigi Rizzo wrote: > On Wed, Sep 25, 2013 at 10:07 AM, hiren panchasara > wrote: > > > > I am providing line rate traffic (via pkg-gen.c) to my 10gig ix > interface. > > > > Now on receiving side, is there a way to sub-divide the traffic into > > multiple workloads using netmap? > > > > For example, can I get two 5G flows from 10Gbps traffic? > > not directly. You'd need to send packets with different addresses that > match > the way the filters on the NIC (RSS or similar) are programmed. > Thanks for quick responses, Liugi. So, FreeBSD needs PF_RING like thing? Any other way we can do it? Someone pointed me to multiqueue bpf too. Not sure if I can use that to achieve what I am looking for. cheers, Hiren From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 09:05:50 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9657157E for ; Wed, 25 Sep 2013 09:05:50 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x232.google.com (mail-lb0-x232.google.com [IPv6:2a00:1450:4010:c04::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 23A5E243D for ; Wed, 25 Sep 2013 09:05:49 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id z5so4923104lbh.9 for ; Wed, 25 Sep 2013 02:05:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=azVs6BPhadee4Ayq5Al/IFt0vAd5ju41EdYaLUVoIjE=; b=IzRzgrisEgjflDbmJDOrRBLhL0b1kxA6x7x9M4T2Lv5CCA5QK4PEPfFPqG8NX5PStf NVZYbMoBpO2ey4R4ieiDXEuo8BysOCa0eEtilkDI92XPhljn4KQ6zIfb7EmfAMnPH9Q0 jR8I2p0FxCWCAutsB+WQBIxkO870N0OQSlJtCbwqb7x9/Li6WyFxjsNyDNasL6HrrdoR u5StByqdBVQ9QdPlITG44ibVUKRHJKT1TadMLrqQLLbkAeVB+p85W/lCEMAiXqz+EO6d zdXg5QdkiyTtcrWNJJykIMNfBhdd7O48k+9PhptDX/APHtC04cFWYUh7FDB0infKj3Pv WxzA== MIME-Version: 1.0 X-Received: by 10.152.30.74 with SMTP id q10mr10960737lah.27.1380099948190; Wed, 25 Sep 2013 02:05:48 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.105 with HTTP; Wed, 25 Sep 2013 02:05:48 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 11:05:48 +0200 X-Google-Sender-Auth: 1vi4sESCJ2IezPvzTBnz8ZwDGUA Message-ID: Subject: Re: netmap: traffic distribution From: Luigi Rizzo To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 09:05:50 -0000 On Wed, Sep 25, 2013 at 10:53 AM, hiren panchasara wrote: > > > > On Wed, Sep 25, 2013 at 1:22 AM, Luigi Rizzo wrote: >> >> On Wed, Sep 25, 2013 at 10:07 AM, hiren panchasara >> wrote: >> > >> > I am providing line rate traffic (via pkg-gen.c) to my 10gig ix >> > interface. >> > >> > Now on receiving side, is there a way to sub-divide the traffic into >> > multiple workloads using netmap? >> > >> > For example, can I get two 5G flows from 10Gbps traffic? >> >> not directly. You'd need to send packets with different addresses that >> match >> the way the filters on the NIC (RSS or similar) are programmed. > > > Thanks for quick responses, Liugi. > > So, FreeBSD needs PF_RING like thing? Any other way we can do it? no, PF_RING does nothing more than netmap. the partitioning of traffic into queues is done by the NIC's hardware, through some filters that i mentioned and are NIC specific. They are often named RSS (receive side scaling), RFS (receive flow steering), Flow Director, and so on. Some NICs compute a hash of various header fields and use the result to direct packets to specific queues. Others have "exact match" filters where you can map certain mac headers to specific queues, and so on. A software demultiplexer that sits on top of a netmap ring may certainly be useful, but i have not yet designed it. cheers luigi From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 09:12:56 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A3153772 for ; Wed, 25 Sep 2013 09:12:56 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x22e.google.com (mail-ee0-x22e.google.com [IPv6:2a00:1450:4013:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 37CAF24D6 for ; Wed, 25 Sep 2013 09:12:56 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id c13so3015478eek.19 for ; Wed, 25 Sep 2013 02:12:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vxQQ0xd4dBk0AlItWPr/M6aL+9kfzY0Xhykqg6nxaxQ=; b=rrggJAhWUIlBQERQXLoChpnyMBLuUtaq26/h+NcHdDbG/3zR3SNhRxguNHOqGmRa9K 3wmn6Q4mb/wpd8B34c831wUWuB2nUjaNVvjR4cBcflxkNbyvnilhfLj9Tco36sB4+Vnw HyX9A7khUX6qSNNpqafe52xtY3WgHMiFdJYjQrl0NPdbf32RjbbXCqh52AtwF+XrGatP bTFQxjH9sd4JldjBYW0lXzbHSWAG5xV3bMat18WVcyon1Xi7iXKKJRL89xFCOEu4K6Zo MPx9J/Y8Ou6SMkETbCqFe8rObXZD6qWo2cgjWdE/SSd8konSegxN4kd7z0wp+GLUW5jO LoOA== MIME-Version: 1.0 X-Received: by 10.14.183.130 with SMTP id q2mr54171440eem.5.1380100374515; Wed, 25 Sep 2013 02:12:54 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Wed, 25 Sep 2013 02:12:54 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 02:12:54 -0700 Message-ID: Subject: Re: netmap: traffic distribution From: hiren panchasara To: Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 09:12:56 -0000 On Wed, Sep 25, 2013 at 2:05 AM, Luigi Rizzo wrote: > On Wed, Sep 25, 2013 at 10:53 AM, hiren panchasara > wrote: > > > > > > > > On Wed, Sep 25, 2013 at 1:22 AM, Luigi Rizzo wrote: > >> > >> On Wed, Sep 25, 2013 at 10:07 AM, hiren panchasara > >> wrote: > >> > > >> > I am providing line rate traffic (via pkg-gen.c) to my 10gig ix > >> > interface. > >> > > >> > Now on receiving side, is there a way to sub-divide the traffic into > >> > multiple workloads using netmap? > >> > > >> > For example, can I get two 5G flows from 10Gbps traffic? > >> > >> not directly. You'd need to send packets with different addresses that > >> match > >> the way the filters on the NIC (RSS or similar) are programmed. > > > > > > Thanks for quick responses, Liugi. > > > > So, FreeBSD needs PF_RING like thing? Any other way we can do it? > > no, > PF_RING does nothing more than netmap. > Okay. > > the partitioning of traffic into queues is done by the NIC's hardware, > through some filters that i mentioned and are NIC specific. > They are often named RSS (receive side scaling), RFS (receive flow > steering), > Flow Director, and so on. Some NICs compute a hash of various header > fields > and use the result to direct packets to specific queues. Others have > "exact match" filters where you can map certain mac headers to > specific queues, and so on. > Alright. I will investigate more about RSS/RFS for ixgbe. Thanks a bunch, Hiren > > A software demultiplexer that sits on top of a netmap ring > may certainly be useful, but i have not yet designed it. > > > cheers > luigi > From owner-freebsd-net@FreeBSD.ORG Wed Sep 25 09:31:38 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CCB79BD6 for ; Wed, 25 Sep 2013 09:31:38 +0000 (UTC) (envelope-from talayeh.asadi@gmail.com) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A079425FE for ; Wed, 25 Sep 2013 09:31:38 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id at1so10488879iec.16 for ; Wed, 25 Sep 2013 02:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:from:date:message-id:subject:to :content-type; bh=drAmQ9HhBC7rdQ3S00KycpVR9FQ50eFtFp5gWn1wfIQ=; b=piKgNxdZ4RjPkY5JLn0GIjFDLLLJBWbcLI3uHPD1t6HH02sA7i25XkuRM2U0Ov0SLj /yXhVrV8yybq9eDcOLX6sByhHxrF+EE+F+Dwq+VqDifAvzdONBkyoytCjXbe1ywwRVib a6BJ9wGAxeP0CPSOLnu94UnEU3ox4sUGWTChcQZ620VaNT75R+7kOd1yx78qBP0aHqxA kgdrSbQFwNiU9BfeS9UHVC2J3oRBxvf8BeDFfiHih+dCRMRuioNTxDSiGon9Q9q6cD+E XvN3FNxiFnhxozIbu+KqStd1cQCyK5tFOd7xr2eBXYfTEKHPFCdM3eMQ/iaV0j1wdqsB BV4A== X-Received: by 10.43.161.3 with SMTP id me3mr106123icc.98.1380101497476; Wed, 25 Sep 2013 02:31:37 -0700 (PDT) MIME-Version: 1.0 Sender: talayeh.asadi@gmail.com Received: by 10.42.153.8 with HTTP; Wed, 25 Sep 2013 02:31:17 -0700 (PDT) From: takCoder Date: Wed, 25 Sep 2013 13:01:17 +0330 X-Google-Sender-Auth: H-aABSAcc_Zw94GdDQGO_8fLDy0 Message-ID: Subject: Suggestions for choosing a good Performance Testing Tool To: Freebsd-net Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: tak.official@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Sep 2013 09:31:38 -0000 Hi everyone, I need and am trying to find a way to run reliable performance tests (RFC2544 test family) on my Network devices(routers, switches, etc).. I am looking for a proper BSD-Based tool, which gives me information about my systems' throughput, latency, packet-drop and alike in the performance test family; I have found some tools, and I'd rather finding an all-in-one solution, also appreciating separate trustworthy tools as well.. as you are network-experts, facing this task more often, would you please share your suggestions with me?? It would be really kind of you t do so.. thank you all in advance.. Best Regards, takCoder From owner-freebsd-net@FreeBSD.ORG Thu Sep 26 00:35:49 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A758EACB; Thu, 26 Sep 2013 00:35:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1F08A2FD7; Thu, 26 Sep 2013 00:35:48 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id cb5so6266788wib.15 for ; Wed, 25 Sep 2013 17:35:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Z8+Ia9gMZiILIjG4fxye8PtTs25Wfjyvz7LkK8JHtfo=; b=AxOdiHxU904g1QOXhM0yu/9EgnW9IGHCQmhJjTN70o4hnFjvINEMpmtqY7WF2VZ656 ZGpbIz5uCpPlwwilUo+valxGOT9YHbvokoZ6Yy4hcqj6nliz5QzlV9t+p2HgHwfArU/7 8MBTOLVpr2/FnXQU32ovyJHkNz3ux8yZRbfxIqMuXmSOVQ4352b6oSrMkwE74l3lEJlr bVTDqg/JGqKB40DXZmOpnxnFLLfuvSFjC0PKI6h39XOdrdo3r2CQYcPbBTl0laMkXWl2 f3y07Agy1EshTBWT3JkYW/+c/cJGv/fTeVEFeZ4NU3Vo09Ia3Wj989N/jOFQOczuevcS OJ1A== MIME-Version: 1.0 X-Received: by 10.194.93.72 with SMTP id cs8mr86877wjb.49.1380155747330; Wed, 25 Sep 2013 17:35:47 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Wed, 25 Sep 2013 17:35:47 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 Sep 2013 17:35:47 -0700 X-Google-Sender-Auth: gClU_wDiITGXi5RetZHUokDPVLg Message-ID: Subject: Re: Exposing sysctls for ixgbe From: Adrian Chadd To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2013 00:35:49 -0000 please cc jfv and get them into his driver. :) -a On 24 September 2013 23:53, hiren panchasara wrote: > Any comments? > > cheers, > Hiren > > On Sun, Sep 22, 2013 at 12:01 PM, hiren panchasara >wrote: > > > $ sysctl hw.igb > > hw.igb.rxd: 4096 > > hw.igb.txd: 4096 > > hw.igb.enable_aim: 1 > > hw.igb.enable_msix: 1 > > hw.igb.max_interrupt_rate: 8000 > > hw.igb.buf_ring_size: 4096 > > hw.igb.header_split: 0 > > hw.igb.num_queues: 1 > > hw.igb.rx_process_limit: 100 > > $ sysctl hw.ix > > sysctl: unknown oid 'hw.ix': No such file or directory > > > > I thought it would be nice to have these things exposed. So I copied them > > from igb: > > http://people.freebsd.org/~hiren/ixgbe_sysctls.txt > > > > Changes for if_igb.c is to expose correct auto-tuned value for a running > > system for "hw.igb.num_queues", which is not the case right now. > > > > Thanks to markj@ for help/pointers. > > > > Please let me know if the diffs look okay. > > > > cheers, > > Hiren > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Sep 26 07:59:39 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3C128B53 for ; Thu, 26 Sep 2013 07:59:39 +0000 (UTC) (envelope-from micchie@outlook.com) Received: from bay0-omc3-s25.bay0.hotmail.com (bay0-omc3-s25.bay0.hotmail.com [65.54.190.163]) by mx1.freebsd.org (Postfix) with ESMTP id 2501925A1 for ; Thu, 26 Sep 2013 07:59:39 +0000 (UTC) Received: from BAY176-W22 ([65.54.190.187]) by bay0-omc3-s25.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 26 Sep 2013 00:58:33 -0700 X-TMN: [eXzTf3KLcw3fh8ujOI6Kj2hg5WyE+T2a] X-Originating-Email: [micchie@outlook.com] Message-ID: From: Michio Honda To: hiren panchasara , Luigi Rizzo Subject: RE: netmap: traffic distribution Date: Thu, 26 Sep 2013 16:58:32 +0900 Importance: Normal In-Reply-To: References: , , , , MIME-Version: 1.0 X-OriginalArrivalTime: 26 Sep 2013 07:58:33.0487 (UTC) FILETIME=[327369F0:01CEBA8E] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2013 07:59:39 -0000 Hi=2C=20 The handiest way to try flexible flow distribution is using Flow Director.I= 've confirmed that the patch posted to this list two years ago works with n= etmap/ixgbe.http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysct= ls-to-ixgbe-4-td4769489.html Cheers=2C- Michio > Date: Wed=2C 25 Sep 2013 02:12:54 -0700 > Subject: Re: netmap: traffic distribution > From: hiren.panchasara@gmail.com > To: rizzo@iet.unipi.it > CC: freebsd-net@freebsd.org >=20 > On Wed=2C Sep 25=2C 2013 at 2:05 AM=2C Luigi Rizzo w= rote: >=20 > > On Wed=2C Sep 25=2C 2013 at 10:53 AM=2C hiren panchasara > > wrote: > > > > > > > > > > > > On Wed=2C Sep 25=2C 2013 at 1:22 AM=2C Luigi Rizzo wrote: > > >> > > >> On Wed=2C Sep 25=2C 2013 at 10:07 AM=2C hiren panchasara > > >> wrote: > > >> > > > >> > I am providing line rate traffic (via pkg-gen.c) to my 10gig ix > > >> > interface. > > >> > > > >> > Now on receiving side=2C is there a way to sub-divide the traffic = into > > >> > multiple workloads using netmap? > > >> > > > >> > For example=2C can I get two 5G flows from 10Gbps traffic? > > >> > > >> not directly. You'd need to send packets with different addresses th= at > > >> match > > >> the way the filters on the NIC (RSS or similar) are programmed. > > > > > > > > > Thanks for quick responses=2C Liugi. > > > > > > So=2C FreeBSD needs PF_RING like thing? Any other way we can do it? > > > > no=2C > > PF_RING does nothing more than netmap. > > > Okay. >=20 > > > > the partitioning of traffic into queues is done by the NIC's hardware= =2C > > through some filters that i mentioned and are NIC specific. > > They are often named RSS (receive side scaling)=2C RFS (receive flow > > steering)=2C > > Flow Director=2C and so on. Some NICs compute a hash of various header > > fields > > and use the result to direct packets to specific queues. Others have > > "exact match" filters where you can map certain mac headers to > > specific queues=2C and so on. > > >=20 > Alright. I will investigate more about RSS/RFS for ixgbe. >=20 > Thanks a bunch=2C > Hiren >=20 > > > > A software demultiplexer that sits on top of a netmap ring > > may certainly be useful=2C but i have not yet designed it. > > > > > > cheers > > luigi > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe=2C send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 = From owner-freebsd-net@FreeBSD.ORG Thu Sep 26 17:38:05 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AEC9E510 for ; Thu, 26 Sep 2013 17:38:05 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x230.google.com (mail-ee0-x230.google.com [IPv6:2a00:1450:4013:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 451562C87 for ; Thu, 26 Sep 2013 17:38:05 +0000 (UTC) Received: by mail-ee0-f48.google.com with SMTP id l10so706265eei.21 for ; Thu, 26 Sep 2013 10:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9z4gWVT0pFnIBoFnWVE+rzTqt++LFwlaYnpEbz/r1wM=; b=xWmIUJ6b4DREE8qvgqTDsgdcuSqbwmoLmSA6yHXH/IAcl28t9jRdP5BUimwgiw1vc0 UughUKTjOxBtrEgcWcrDhyy+eUzJtw4rEXGrA7cBr9Kvn34PEFXfiSil7UH+TBCwD7RY 1jAeEzTASZg2eDYmwh/4tC7ArzREO5BjitVGKhv0nO7AX9wgQxGE9MpzNcmuDBtg0SWA qybsehZgD7zGZ7YMCvl6tsfPr5RNsfXq9zbfNIF/AjIrXOrEaLFEkP14NVdyquqD2aKu bo+q/5ECGKjoMSzaZ+6QCgTWVenNSYHrBMjrdVKFaJEnFsQtnTuQZeEuHcGsNygGxt46 u8bQ== MIME-Version: 1.0 X-Received: by 10.15.33.132 with SMTP id c4mr3107494eev.2.1380217083517; Thu, 26 Sep 2013 10:38:03 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Thu, 26 Sep 2013 10:38:03 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Sep 2013 10:38:03 -0700 Message-ID: Subject: Re: netmap: traffic distribution From: hiren panchasara To: Michio Honda Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2013 17:38:05 -0000 On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda wrote: > Hi, > > The handiest way to try flexible flow distribution is using Flow Director. > I've confirmed that the patch posted to this list two years ago works with > netmap/ixgbe. > > http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html > Thanks a lot for the link, Michio! It seems this work is yet not committed?!? cheers, Hiren > > Cheers, > - Michio > From owner-freebsd-net@FreeBSD.ORG Thu Sep 26 20:21:16 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D7949714 for ; Thu, 26 Sep 2013 20:21:16 +0000 (UTC) (envelope-from garmitage@swin.edu.au) Received: from gpo1.cc.swin.edu.au (gpo1.cc.swin.edu.au [136.186.1.30]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 55D1128A7 for ; Thu, 26 Sep 2013 20:21:15 +0000 (UTC) Received: from [136.186.229.37] (garmitage.caia.swin.edu.au [136.186.229.37]) by gpo1.cc.swin.edu.au (8.14.3/8.14.3) with ESMTP id r8QKKdES019957 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 27 Sep 2013 06:21:00 +1000 Message-ID: <52449717.6060605@swin.edu.au> Date: Fri, 27 Sep 2013 06:20:39 +1000 From: grenville armitage User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121107 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: New: FreeBSD TCP stack inside NS3 simulations Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2013 20:21:16 -0000 All, Hopefully of interest to this list (and apologies in advanced if you see this on multiple mailing lists). I'm please to announce the v0.1 release of our NS-3/NSC environment for doing network traffic simulations, using the actual FreeBSD TCP stack inside the simulations. The tool itself has been released as a VirtualBox virtual machine (VM) appliance. This VM provides a FreeBSD 9-based turn-key environment to experiment with the CAIA NS-3/NSC modifications. Patched versions of the NS-3 and NSC code are provided, along with ready-to-compile-and-run example simulation code (multi-leaf dumbbell topology and incast topology using FreeBSD 9's TCP stack in the simulated end hosts). Simple scripts to produce initial plots of simulation output are also provided as a starting point. The VirtualBox ova can be found at http://caia.swin.edu.au/urp/incast/tools.html Many more details can be found in the README at http://caia.swin.edu.au/urp/incast/tools/README.caia-freebsd9-amd64-caia-ns3-release-0.1.txt This work is the culmination of significant effort by Lawrence Stewart (lstewart@freebsd.org) as part of his PhD work, and was supported in part by a gift from the Cisco University Research Program Fund. cheers, gja -- Professor Grenville Armitage Director, Centre for Advanced Internet Architectures Faculty of Information and Communication Technologies Swinburne University of Technology, Australia http://caia.swin.edu.au From owner-freebsd-net@FreeBSD.ORG Thu Sep 26 21:27:48 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 41F9D788 for ; Thu, 26 Sep 2013 21:27:48 +0000 (UTC) (envelope-from rometoroam@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CED4E2C27 for ; Thu, 26 Sep 2013 21:27:47 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id l18so23353wgh.4 for ; Thu, 26 Sep 2013 14:27:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ynn4AEkyKUxzvTGJEJhMXWeAz5PNUGUEZzqfQg/7d34=; b=fBdMXL1Mxumqt9jsc4PcPrXl3m6UB6FjaKvOTZI5PJQCasQocWmTo8OlgOiub4Xbpk 0ekjN93tJrppCZEMe65LJtVzD+1QxZY0N//L7GVTdlrLc81SDhchaRpguX+t3dJJaFyr FyS4SjxUl8vMPjjcdTrzvab89NFBEegwYsjwILTmE7eAQeRGANJbKGm2RdmZiev/DJqx hqUf7js7wZZwQXJmDLnP+r8/Uw0E1JSGPb5lgMWIdjP0c5gO5RkMSVr4NghxUxQqkPTq TXTn7lLLt190TfeM5m6oVIFvW/l5Lm5YHUR6P+qcuM6n8MeSyMx5KzOm7d8QV28asqR/ ieSg== MIME-Version: 1.0 X-Received: by 10.180.171.7 with SMTP id aq7mr2601501wic.28.1380230865855; Thu, 26 Sep 2013 14:27:45 -0700 (PDT) Received: by 10.194.64.167 with HTTP; Thu, 26 Sep 2013 14:27:45 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Sep 2013 17:27:45 -0400 Message-ID: Subject: Re: netmap: traffic distribution From: chintu hetam To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Michio Honda , Luigi Rizzo , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Sep 2013 21:27:48 -0000 Hiren, https://www.kernel.org/doc/Documentation/networking/scaling.txt must read to understand nuances of each of this features. None of this techniques are used for mostly none other than performance reasons. Michio, personally i am interested to know performance results in netmap mode with RFS patch you just mentioned. Thanks r2r On Thu, Sep 26, 2013 at 1:38 PM, hiren panchasara < hiren.panchasara@gmail.com> wrote: > On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda > wrote: > > > Hi, > > > > The handiest way to try flexible flow distribution is using Flow > Director. > > I've confirmed that the patch posted to this list two years ago works > with > > netmap/ixgbe. > > > > > http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html > > > > Thanks a lot for the link, Michio! > > It seems this work is yet not committed?!? > > cheers, > Hiren > > > > > Cheers, > > - Michio > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 06:59:07 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C84478C6 for ; Fri, 27 Sep 2013 06:59:07 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x22a.google.com (mail-ee0-x22a.google.com [IPv6:2a00:1450:4013:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5DFCD2653 for ; Fri, 27 Sep 2013 06:59:07 +0000 (UTC) Received: by mail-ee0-f42.google.com with SMTP id b45so1003337eek.15 for ; Thu, 26 Sep 2013 23:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aDQw/nbJMR0hqnQJHYctFlfmnVM8pCnb447zN+tHRdg=; b=WuDAKJ0X2xONV0KzaqcYo53v76Uta1MbobuLzxwmS/fejC4zk5uLVPMrYwydit1RCL JokV00dEiOvcx/GswU3CQY8QM2+039AD2+hGyjKnYXxdiH8zZ+PPkX9ur7105OFZUOer bV6vqMYqdHqaBU+GIIJuPVYOSROIXaYv0wB3SuDhSHRp7pBs8Q/PtdF8E2y0SeLRJWbt MdmBoh59ELhsqZuieqSBgNWkTG+0eNBjxlROQjWG+QrZ3KZacUx6+o6n24UNjvZBcMNn ZxXg9oGWwam0hfMh6pVQ6tvFLrlJnvROFOwmzj/CP8q/F9avewViBzyKRW19rKBz1PiV NaSA== MIME-Version: 1.0 X-Received: by 10.14.108.9 with SMTP id p9mr8137286eeg.8.1380265145851; Thu, 26 Sep 2013 23:59:05 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Thu, 26 Sep 2013 23:59:05 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Sep 2013 23:59:05 -0700 Message-ID: Subject: Re: netmap: traffic distribution From: hiren panchasara To: chintu hetam Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Michio Honda , Luigi Rizzo , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 06:59:07 -0000 On Thu, Sep 26, 2013 at 2:27 PM, chintu hetam wrote: > Hiren, > > https://www.kernel.org/doc/Documentation/networking/scaling.txt must read > to understand nuances of each of this features. None of this techniques are > used for mostly none other than performance reasons. > Thanks for the link. So, RFS (Receive Flow Steering) is equivalent to "flow director" mentioned in FreeBSD's ixgbe drivers? > > Michio, personally i am interested to know performance results in netmap > mode with RFS patch you just mentioned. > Takuya/Luigi might have some numbers. Thanks, Hiren From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 07:43:19 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2F443366 for ; Fri, 27 Sep 2013 07:43:19 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x22a.google.com (mail-ee0-x22a.google.com [IPv6:2a00:1450:4013:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BF9452892 for ; Fri, 27 Sep 2013 07:43:18 +0000 (UTC) Received: by mail-ee0-f42.google.com with SMTP id b45so1028631eek.15 for ; Fri, 27 Sep 2013 00:43:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=U+QSZ/+rBzc88Pr16AjvcTciMtnD/bPDHO+bawhk8Pk=; b=J5SBfGIcLLJT6JRj+XNdSgY00XqgbKr8UzUKPwEoKB+ZIcBINbpo19BMt0W0qHKnde UsQSfeKcK2rB/C0AW42/rAVVMVWKHVE7+CW32Ia2g+WXHs0Avf74xj0bLghyIDbJ/fZy MhsO6D+SAZwZ2TdGkALQ25SA7ZMyxCSl/7Sdp2La7e6EQHiotcrX+LEBqhckhrU7CZvL ETWE4TMbk2iz1VJSg3ijxi+roKK2uXStJ89CGnWpTOsGWycH7v5wu+goQjMQz3kmpBk4 lEKXqBaYvyMBvQ+9HAB0hb07YTIibMmlsEVSTNAZyykwAxUCd69RL0sN5EQv5IYC+xLn LLFw== MIME-Version: 1.0 X-Received: by 10.14.5.3 with SMTP id 3mr8311078eek.49.1380267797247; Fri, 27 Sep 2013 00:43:17 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Fri, 27 Sep 2013 00:43:17 -0700 (PDT) Date: Fri, 27 Sep 2013 00:43:17 -0700 Message-ID: Subject: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) From: hiren panchasara To: "freebsd-net@freebsd.org" , syuu@dokukino.com Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 07:43:19 -0000 On Thu, Sep 26, 2013 at 10:38 AM, hiren panchasara < hiren.panchasara@gmail.com> wrote: > > > > On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda wrote: > >> Hi, >> >> The handiest way to try flexible flow distribution is using Flow Director. >> I've confirmed that the patch posted to this list two years ago works >> with netmap/ixgbe. >> >> http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html >> > > Thanks a lot for the link, Michio! > > It seems this work is yet not committed?!? > Takuya, I see a lot of responses/comments on proposed changes. Was anything decided at the end of it? As far as I can tell, its still not committed to the tree. Thanks, Hiren From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 08:29:42 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4391A26D for ; Fri, 27 Sep 2013 08:29:42 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D42172AF9 for ; Fri, 27 Sep 2013 08:29:41 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hn9so453074wib.5 for ; Fri, 27 Sep 2013 01:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Z6qn3hK5RZSY+3AFrm6VlhvSbzL08+uCov8168ipXBU=; b=c44sh5uoDYYz2H7SP1Of9A+6ymYTBu4C4mqv9qmGdFyRT578ZicsF5aOJwgAWIIPIM HXabPx4xQ2nwkYWFvePsVpOCCK96ep2cuN4SR+uZXbJQAfFt5YzzWpw4YqXz+/5Cj/8r Vt51+MJP5OQ6NOZEU/WjfVsVBp6QhI//d2v1Vwd8mg7dkBUPNNkHxGoVfo5s+VSU5Mcj Hs7PVIUvTRqgF8E5uWlFIK1dQoh01XchayuzvJhfHEpzUZ/j2dbc09ID/+QD+ZvoGsGS 8+QO/xzXSkyA54Q1gsbYtWtDYSx6i3a0nLtk8n24x79VGEbb9mFIrfk9oroSKbpqsGvk d0GQ== MIME-Version: 1.0 X-Received: by 10.180.75.236 with SMTP id f12mr1666650wiw.0.1380270580253; Fri, 27 Sep 2013 01:29:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Fri, 27 Sep 2013 01:29:40 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 01:29:40 -0700 X-Google-Sender-Auth: fwwTbGaayTbmoN04Gk8-6WR4vSg Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) From: Adrian Chadd To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Takuya ASADA , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:29:42 -0000 On 27 September 2013 00:43, hiren panchasara wrote: > Takuya, > > I see a lot of responses/comments on proposed changes. Was anything decided > at the end of it? As far as I can tell, its still not committed to the > tree. > I'd rather see an ioctl API for that chipset and then have a separate tool program it for now. So, how bout we hack that up? :) -adrian From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 08:40:02 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 558F6556 for ; Fri, 27 Sep 2013 08:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4305D2B7A for ; Fri, 27 Sep 2013 08:40:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r8R8e1N9016528 for ; Fri, 27 Sep 2013 08:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r8R8e12u016527; Fri, 27 Sep 2013 08:40:01 GMT (envelope-from gnats) Date: Fri, 27 Sep 2013 08:40:01 GMT Message-Id: <201309270840.r8R8e12u016527@freefall.freebsd.org> To: freebsd-net@FreeBSD.org Cc: From: Eugene Grosbein Subject: Re: kern/172113: [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4): m_getjcl: invalid cluster type X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Eugene Grosbein List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:40:02 -0000 The following reply was made to PR kern/172113; it has been noted by GNATS. From: Eugene Grosbein To: bug-followup@FreeBSD.ORG Cc: Jack Vogel , George Neville-Neil , John Baldwin Subject: Re: kern/172113: [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4): m_getjcl: invalid cluster type Date: Fri, 27 Sep 2013 15:31:58 +0700 Hi! Audit-Trail of http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/172113 has more than enough patches to fix the problem. Would someone be so kind to commit one of them so 9.2-RELEASE won't be broken out-of-the-box for noted Supermicro's? From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 08:50:06 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DA26285F for ; Fri, 27 Sep 2013 08:50:06 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-pd0-x22e.google.com (mail-pd0-x22e.google.com [IPv6:2607:f8b0:400e:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AF2042BFC for ; Fri, 27 Sep 2013 08:50:06 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id y13so2317772pdi.19 for ; Fri, 27 Sep 2013 01:50:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dokukino.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ohZ5V4/sljQ6/Csy1FSdAbtVytY5aJDPg7Oa0UyVN+I=; b=SdgeDjBslWDV9eh7EkGj0iaZR4Bd05nfjHFDst4AUEtRJJUhC2rltmjRBCRtTY9LoU QWlNGyQRJOoB7TWGk4lhJI4H1y2Ozq3wvoOshPwQKQnwNWBB/Wz5c5J17h8CsZpI5mlg QI6FEUqstQ312LLxo31FjI1Mr7bqMFf081Ad4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=ohZ5V4/sljQ6/Csy1FSdAbtVytY5aJDPg7Oa0UyVN+I=; b=PJxAqAjHQskEhaGMuBT8UtRzdLf8WseyhlWu7A1uw5VkS1S4+r0WIJvvBGVyOA56e2 MQu89hT+yyHhM6lQMku0S7jSCpdApQDAZp2wcWHcTkeaJIjBforXSEtJuqI++DaLcTql vsg9vSl+iY9jk6aTiVQdyAhoM/mD5Fi0u1IApXC/vlXUbPrJKr49w4/RBohiiY/ssfRY UAaIsyqH08ORS4Q3/sacveXt1UtXBIkvgD2A7JVitrf7jKRIppZnizkqJ6L5Mvzw/cz3 h0UMLOrVUGqV5Vygpd/2t8hd6D3Ta3tfSmTGbp9Ob7zTuMEUaza63oybDxqO/VKi1l4Q C1ng== X-Gm-Message-State: ALoCoQlVIWYisYdwY+QGApLttint5CkHSnSJh2dkJggo6yuaVBsvUchTcZnG8jxmaQjMtXI4nnRn X-Received: by 10.66.102.40 with SMTP id fl8mr10246273pab.167.1380271805513; Fri, 27 Sep 2013 01:50:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.220.138 with HTTP; Fri, 27 Sep 2013 01:49:24 -0700 (PDT) In-Reply-To: References: From: Takuya ASADA Date: Fri, 27 Sep 2013 17:49:24 +0900 Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) To: hiren panchasara Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:50:06 -0000 Hi, I think there were many comment which says "don't use device specific sysctls, we should have more generic interface to configure NIC filter, like Linux's ethtool". And I heard same idea in last BSDCan, but there's still no proposal of "generic interface for NIC filter", I suppose. So, honestly I haven't good idea to merge the change. Is there any people who has good idea to handle this? Or, just merge this patch would be fine? 2013/9/27 hiren panchasara > > > > On Thu, Sep 26, 2013 at 10:38 AM, hiren panchasara < > hiren.panchasara@gmail.com> wrote: > >> >> >> >> On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda wrote: >> >>> Hi, >>> >>> The handiest way to try flexible flow distribution is using Flow >>> Director. >>> I've confirmed that the patch posted to this list two years ago works >>> with netmap/ixgbe. >>> >>> http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html >>> >> >> Thanks a lot for the link, Michio! >> >> It seems this work is yet not committed?!? >> > > Takuya, > > I see a lot of responses/comments on proposed changes. Was anything > decided at the end of it? As far as I can tell, its still not committed to > the tree. > > Thanks, > Hiren > > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 08:53:04 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B566CA0A for ; Fri, 27 Sep 2013 08:53:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 36D442C4B for ; Fri, 27 Sep 2013 08:53:04 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id b13so2375798wgh.35 for ; Fri, 27 Sep 2013 01:53:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=lmv+StQ09gVhHfnk+TD/QdyxGbA1S0Xm2Q2pc9cgYjk=; b=L5UPtZAtFFgV5JQUc6pFM2OIXNHc/eektZtC7fkf4RYeojjx6/Q//G/GbncPPjeM+Y NkPWVh9exT53UYKYbWeYV95TthdH1XFNFvJk0fOLSumSfBI48bbn9UWN8vPl1p0CUdGs +fwwEYf1QxctTmr8hTzVBzbb0tAu6Sa6OBBwdDiqLsv4uhTB7prbU1HVehPCOLc3+boj PYwyC8aFF53CIav+NdXVpJl+qiurLISTXNmI9O8HOqJ3sR1B+ppvI9KSIMrEBQ0OgCmV +13XT3SQUqwtFd5V5CDmbkMevkzb0fvq1rtzkJsKJMf1ifAN8lxeKkRDoFivpbtHwNch GY/w== MIME-Version: 1.0 X-Received: by 10.180.75.236 with SMTP id f12mr1748353wiw.0.1380271982512; Fri, 27 Sep 2013 01:53:02 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Fri, 27 Sep 2013 01:53:02 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 01:53:02 -0700 X-Google-Sender-Auth: EYGwVQPZ20hna4_ju50TQIEBLMM Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) From: Adrian Chadd To: Takuya ASADA Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , hiren panchasara X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:53:04 -0000 I don't care about whether there's a generic API right now. I'd rather see it done as a staged thing, but _not_ sysctls. Having sysctls to add/remove entries from things is just plain evil. I'd rather instead come up with a device specific ioctl API for this for now w/ a userland tool for each particular chip. Then once we all get a bit more experience doing this, a unified API can be proposed. -adrian On 27 September 2013 01:49, Takuya ASADA wrote: > Hi, > > I think there were many comment which says "don't use device specific > sysctls, we should have more generic interface to configure NIC filter, > like Linux's ethtool". > And I heard same idea in last BSDCan, but there's still no proposal of > "generic interface for NIC filter", I suppose. > So, honestly I haven't good idea to merge the change. > Is there any people who has good idea to handle this? > > Or, just merge this patch would be fine? > > > 2013/9/27 hiren panchasara > > > > > > > > > On Thu, Sep 26, 2013 at 10:38 AM, hiren panchasara < > > hiren.panchasara@gmail.com> wrote: > > > >> > >> > >> > >> On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda >wrote: > >> > >>> Hi, > >>> > >>> The handiest way to try flexible flow distribution is using Flow > >>> Director. > >>> I've confirmed that the patch posted to this list two years ago works > >>> with netmap/ixgbe. > >>> > >>> > http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html > >>> > >> > >> Thanks a lot for the link, Michio! > >> > >> It seems this work is yet not committed?!? > >> > > > > Takuya, > > > > I see a lot of responses/comments on proposed changes. Was anything > > decided at the end of it? As far as I can tell, its still not committed > to > > the tree. > > > > Thanks, > > Hiren > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 08:59:24 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DEAFDBB0 for ; Fri, 27 Sep 2013 08:59:24 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-pb0-x236.google.com (mail-pb0-x236.google.com [IPv6:2607:f8b0:400e:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id ACB902C95 for ; Fri, 27 Sep 2013 08:59:24 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id ro12so2290970pbb.13 for ; Fri, 27 Sep 2013 01:59:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dokukino.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=Jx4sMa5XSV+98SrGcQM3zrqZixa0rKIXRKka3XWqobE=; b=v3x0eFDH1ktTEGt1IKFHOXkuN0x0miUVrlbv8HYopdbHaSwIpPxJxaVi4rR4JRkzt1 S9HktsDisiqiw9/dUUL9UPpIru+ITxcqMdHPfYaWYwI691FUfu3d1DKXukSgfKX0RL5A k0rX9QEbR8HkiM5uRNBTuncrPzqU6AiSqxrAA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=Jx4sMa5XSV+98SrGcQM3zrqZixa0rKIXRKka3XWqobE=; b=nBfoWyRkFZ5Ngk7Ox5S0/31xtmAtjQUOho41YQBSnuf+r9NjiCSq4K8In6dd2u+QPj mCqmEb06gF4Q/p4JPFjCFBXUwzXmoBZalMutZ/UUF6cbDNKKbIykxJ2ndzwIzMbawHPY 5LG4yIWu5CwzLD6R/JhFEIMA71z9w/lUYxfwYox6Y1sZIGWyWGZN5lAk4NRmyRShSURs WK0F7Gn4587Gg9W/21yzj75VRKxFkcKFevZbiEkWir1pG46W+xNMcVNDoXLZwDcgWgvt H3OUOr4Iy8fquRzFjsLuy6+14hwolFeXi9CNpfcHnztPtf0nwNA2hCi4/LJWzmoPkfgY D3AA== X-Gm-Message-State: ALoCoQmNcPZH/h1iWoIrCIXSArIznnD7dZhoHwDBYSy7WkBbCiAVnZcJ8IMhnSav5180jRMvK9eb X-Received: by 10.67.1.228 with SMTP id bj4mr10006746pad.157.1380272363839; Fri, 27 Sep 2013 01:59:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.220.138 with HTTP; Fri, 27 Sep 2013 01:58:43 -0700 (PDT) In-Reply-To: References: From: Takuya ASADA Date: Fri, 27 Sep 2013 17:58:43 +0900 Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , hiren panchasara X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 08:59:24 -0000 2013/9/27 Adrian Chadd > On 27 September 2013 00:43, hiren panchasara wrote: > > >> Takuya, >> >> I see a lot of responses/comments on proposed changes. Was anything >> decided >> at the end of it? As far as I can tell, its still not committed to the >> tree. >> > > I'd rather see an ioctl API for that chipset and then have a separate tool > program it for now. > Ah, like cxgbetool and cxgbe? (it has device specific tool and ioctls) http://svnweb.freebsd.org/base/head/tools/tools/cxgbetool/ http://svnweb.freebsd.org/base/head/sys/dev/cxgb/ > So, how bout we hack that up? :) > Sound's interesting ;-) Could you tell me more detail about your idea? From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 09:00:06 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 87FD9D4B for ; Fri, 27 Sep 2013 09:00:06 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 286D02CBA for ; Fri, 27 Sep 2013 09:00:06 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hn9so488147wib.5 for ; Fri, 27 Sep 2013 02:00:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=+wf7ZNEeLCM7T9UUg49w4jLwsnCHqhmnf/vr4xQ4OHQ=; b=GJc1YIVrBj7BFezn4+oCDxktrHllY4W4itWy1zRkzmCFwXfv+OB8eN5vGzmcu4nWMw 4fPeI9VZ4zXVlohxsK8pHctGmHnFZq69vJy/2E1behcz+13rmpJ1o2Oaad+/ePcLNXmH JS2S2A7N9CarJ7vPEwlfSzivpnQOUa1UZhvGh25WubX3Em0lSv7QTaXFUDrK3BvH4Wsq 25KNMby10EX/jZDERVRs3MQ8CYikTBW3s8C7fq+KeGJK0pBsNXNA/QAEgauCe2pwA6ZP u4Z6dTf39noBsj1MMLQZLq8vK5Jek/RhoVyQysTlRoJeB42iDZfNPz32u3tLvp4N1gBg t45Q== MIME-Version: 1.0 X-Received: by 10.181.12.75 with SMTP id eo11mr1710675wid.24.1380272404621; Fri, 27 Sep 2013 02:00:04 -0700 (PDT) Received: by 10.216.62.5 with HTTP; Fri, 27 Sep 2013 02:00:04 -0700 (PDT) In-Reply-To: <201309270840.r8R8e12u016527@freefall.freebsd.org> References: <201309270840.r8R8e12u016527@freefall.freebsd.org> Date: Fri, 27 Sep 2013 13:00:04 +0400 Message-ID: Subject: Re: kern/172113: [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4): m_getjcl: invalid cluster type From: Sergey Kandaurov Cc: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 09:00:06 -0000 I forward these mails for archives to easier track what was committed for this PR. ---------- Forwarded message ---------- From: Jack F Vogel Date: 6 August 2013 22:00 Subject: svn commit: r254002 - head/sys/dev/e1000 To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Author: jfv Date: Tue Aug 6 18:00:53 2013 New Revision: 254002 URL: http://svnweb.freebsd.org/changeset/base/254002 Log: When the igb driver is static there are cases when early interrupts occur, resulting in a panic in refresh_mbufs, to prevent this add a check in the interrupt handler for DRV_RUNNING. MFC after: 1 day (critical for 9.2) Modified: head/sys/dev/e1000/if_igb.c Modified: head/sys/dev/e1000/if_igb.c ============================================================================== --- head/sys/dev/e1000/if_igb.c Tue Aug 6 17:22:06 2013 (r254001) +++ head/sys/dev/e1000/if_igb.c Tue Aug 6 18:00:53 2013 (r254002) @@ -1572,6 +1572,10 @@ igb_msix_que(void *arg) u32 newitr = 0; bool more_rx; + /* Ignore spurious interrupts */ + if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) + return; + E1000_WRITE_REG(&adapter->hw, E1000_EIMC, que->eims); ++que->irqs; ---------- Forwarded message ---------- From: Jack F Vogel Date: 7 August 2013 01:16 Subject: svn commit: r254009 - releng/9.2/sys/dev/e1000 To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-releng@freebsd.org Author: jfv Date: Tue Aug 6 21:16:00 2013 New Revision: 254009 URL: http://svnweb.freebsd.org/changeset/base/254009 Log: When the igb driver is static there are cases when early interrupts occur, resulting in a panic in refresh_mbufs, to prevent this add a check to the interrupt handler for DRV_RUNNING. Approved by: re Modified: releng/9.2/sys/dev/e1000/if_igb.c Directory Properties: releng/9.2/sys/ (props changed) releng/9.2/sys/dev/ (props changed) releng/9.2/sys/dev/e1000/ (props changed) Modified: releng/9.2/sys/dev/e1000/if_igb.c ============================================================================== --- releng/9.2/sys/dev/e1000/if_igb.c Tue Aug 6 21:01:38 2013 (r254008) +++ releng/9.2/sys/dev/e1000/if_igb.c Tue Aug 6 21:16:00 2013 (r254009) @@ -1572,6 +1572,10 @@ igb_msix_que(void *arg) u32 newitr = 0; bool more_rx; + /* Ignore spurious interrupts */ + if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) + return; + E1000_WRITE_REG(&adapter->hw, E1000_EIMC, que->eims); ++que->irqs; From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 09:01:57 2013 Return-Path: Delivered-To: freebsd-net@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 701A3DF9; Fri, 27 Sep 2013 09:01:57 +0000 (UTC) (envelope-from pluknet@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 43B212CF9; Fri, 27 Sep 2013 09:01:57 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r8R91v4m021866; Fri, 27 Sep 2013 09:01:57 GMT (envelope-from pluknet@freefall.freebsd.org) Received: (from pluknet@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r8R91v5l021865; Fri, 27 Sep 2013 09:01:57 GMT (envelope-from pluknet) Date: Fri, 27 Sep 2013 09:01:57 GMT Message-Id: <201309270901.r8R91v5l021865@freefall.freebsd.org> To: egrosbein@rdtc.ru, pluknet@FreeBSD.org, freebsd-net@FreeBSD.org From: pluknet@FreeBSD.org Subject: Re: kern/172113: [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4): m_getjcl: invalid cluster type X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 09:01:57 -0000 Synopsis: [panic] [e1000] [patch] 9.1-RC1/amd64 panices in igb(4): m_getjcl: invalid cluster type State-Changed-From-To: open->patched State-Changed-By: pluknet State-Changed-When: Fri Sep 27 09:00:31 UTC 2013 State-Changed-Why: The fix committed to HEAD (r254002), stable/9 (r254003), releng/9.2 (r254009). http://www.freebsd.org/cgi/query-pr.cgi?pr=172113 From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 10:09:27 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EE7AC2A0; Fri, 27 Sep 2013 10:09:26 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B4F6B2090; Fri, 27 Sep 2013 10:09:26 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-245-177.lns20.per2.internode.on.net [121.45.245.177]) (authenticated bits=0) by vps1.elischer.org (8.14.6/8.14.6) with ESMTP id r8RA9IaV039410 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 27 Sep 2013 03:09:22 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <52455949.5070501@freebsd.org> Date: Fri, 27 Sep 2013 18:09:13 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: Adding Flow Director sysctls to ixgbe(4) References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Takuya ASADA , "freebsd-net@freebsd.org" , hiren panchasara X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 10:09:27 -0000 On 9/27/13 4:53 PM, Adrian Chadd wrote: > I don't care about whether there's a generic API right now. I'd rather see > it done as a staged thing, but _not_ sysctls. > > Having sysctls to add/remove entries from things is just plain evil. > > I'd rather instead come up with a device specific ioctl API for this for > now w/ a userland tool for each particular chip. Then once we all get a bit > more experience doing this, a unified API can be proposed. that makes it worse If you want to put a device specific sysctl/ioctl set out there then have a device INDEPENDENT tool that knows how to handle the devices we have modified and when we have enough examples we can change the ioctl/sysctl interface to a generic one without changing the interface people are using in their scripts. > > > -adrian > > > > On 27 September 2013 01:49, Takuya ASADA wrote: > >> Hi, >> >> I think there were many comment which says "don't use device specific >> sysctls, we should have more generic interface to configure NIC filter, >> like Linux's ethtool". >> And I heard same idea in last BSDCan, but there's still no proposal of >> "generic interface for NIC filter", I suppose. >> So, honestly I haven't good idea to merge the change. >> Is there any people who has good idea to handle this? >> >> Or, just merge this patch would be fine? >> >> >> 2013/9/27 hiren panchasara >> >>> >>> >>> On Thu, Sep 26, 2013 at 10:38 AM, hiren panchasara < >>> hiren.panchasara@gmail.com> wrote: >>> >>>> >>>> >>>> On Thu, Sep 26, 2013 at 12:58 AM, Michio Honda >> wrote: >>>>> Hi, >>>>> >>>>> The handiest way to try flexible flow distribution is using Flow >>>>> Director. >>>>> I've confirmed that the patch posted to this list two years ago works >>>>> with netmap/ixgbe. >>>>> >>>>> >> http://freebsd.1045724.n5.nabble.com/Adding-Flow-Director-sysctls-to-ixgbe-4-td4769489.html >>>> Thanks a lot for the link, Michio! >>>> >>>> It seems this work is yet not committed?!? >>>> >>> Takuya, >>> >>> I see a lot of responses/comments on proposed changes. Was anything >>> decided at the end of it? As far as I can tell, its still not committed >> to >>> the tree. >>> >>> Thanks, >>> Hiren >>> >>> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 13:10:58 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9147ED55; Fri, 27 Sep 2013 13:10:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 06CA12C33; Fri, 27 Sep 2013 13:10:57 +0000 (UTC) Received: by mail-wg0-f48.google.com with SMTP id n12so2658954wgh.15 for ; Fri, 27 Sep 2013 06:10:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=CnHqp5m+uzPFDFMt7aPvnGuk5vdXRuB6OA9koDoZbLw=; b=k0krTpbp+Be9HvP36MFuZx4POU64SQUT+jbj33NwqfEzH8RU502hE+J2UVGTh/LBMg 3x/kdk6Se0K9qe3Z62c6gUWWvUqHhaKjPTjsL6BVUIiHF0o3n7QoAZZDJUyJBnRaMDuo a5sPAoYsIaOfVCT1JS3F7y524IEMbnqcK8y3vW682ODyIiXW+UsovNPI7FKWy9Ek0Qfk fl50SBbU5XumF5S5QUnbiJ/Pz7jjUqVAOunwobVL7Foz50vhSEKnIn3sxoP8BelvQWls iMly4JalrrXFSnO7MDXxXYPeZBdjPE0b0GKDhqQItRgm5eVLwLs4Hio0uHCWzFoFmhOw IJUA== MIME-Version: 1.0 X-Received: by 10.180.10.136 with SMTP id i8mr2612895wib.46.1380287456362; Fri, 27 Sep 2013 06:10:56 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.73.133 with HTTP; Fri, 27 Sep 2013 06:10:56 -0700 (PDT) In-Reply-To: <52455949.5070501@freebsd.org> References: <52455949.5070501@freebsd.org> Date: Fri, 27 Sep 2013 06:10:56 -0700 X-Google-Sender-Auth: OxbRwQwCIubvn5Z5AJmEsQjUPJY Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) From: Adrian Chadd To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Takuya ASADA , "freebsd-net@freebsd.org" , hiren panchasara X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 13:10:58 -0000 On 27 September 2013 03:09, Julian Elischer wrote: > On 9/27/13 4:53 PM, Adrian Chadd wrote: > >> I don't care about whether there's a generic API right now. I'd rather see >> it done as a staged thing, but _not_ sysctls. >> >> Having sysctls to add/remove entries from things is just plain evil. >> >> I'd rather instead come up with a device specific ioctl API for this for >> now w/ a userland tool for each particular chip. Then once we all get a >> bit >> more experience doing this, a unified API can be proposed. >> > > that makes it worse > If you want to put a device specific sysctl/ioctl set out there then have > a device INDEPENDENT > tool that knows how to handle the devices we have modified and when we > have enough examples we can change the > ioctl/sysctl interface to a generic one without changing the interface > people are using in their scripts. > I agree that's the eventual goal, sure. But there's not necessarily a clear-cut set of shared behaviour that a generic API could actually use. That's why I'm a fan of doing it in a couple of stages - get the device-specific stuff into the tree, get that stuff debugged, then sit down in the future and figure out what the sane intersection of this. The shared API may be partly "shared overlapping behaviour"and may partly be "have a capabilities api that defines which parts of the behaviour is supported by the NIC." Thanks, -adrian From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 15:52:11 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0A7E9E79 for ; Fri, 27 Sep 2013 15:52:11 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ee0-x229.google.com (mail-ee0-x229.google.com [IPv6:2a00:1450:4013:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 84EDD2A56 for ; Fri, 27 Sep 2013 15:52:10 +0000 (UTC) Received: by mail-ee0-f41.google.com with SMTP id d17so1321121eek.28 for ; Fri, 27 Sep 2013 08:52:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=OBX3hZkENbY53Yxnqm2wryV1D0loSko9soaCDB8nUzc=; b=NTVK5VduZXD2NYPa0aP4wXSuB6BBAmVZX8EHQZajAUNP9lZaN+d0EfyuBjSoYv/cbG 7aObNwa4+LHJkOHqrbt0WJY4MrgzLlZZ47XqeEmSwAUUj87aSyo0VALM4u4Dhgom4LCi hzc+PfPlw0tYhMQ/M2M20ZEHeCdOaCII/0AVBz72fCA+kDsdYoiAFptbJrQitKPizPVC bGf1DSoTemPJl3AbAnWdHWjVFtFdVXybHyUbtE80ktMfl7n0m5+jPzSWA4kVsiKVYaa+ kIQjYBnywm5sGAcgbyeGvPrughoxcpchfq46X+Hy26nGj3wHh4nsQa3hms2tvnQ04SJ5 R/lQ== MIME-Version: 1.0 X-Received: by 10.15.33.132 with SMTP id c4mr11522740eev.2.1380297128903; Fri, 27 Sep 2013 08:52:08 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Fri, 27 Sep 2013 08:52:08 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 08:52:08 -0700 Message-ID: Subject: Re: netmap: traffic distribution From: hiren panchasara To: chintu hetam , Michio Honda , "freebsd-net@freebsd.org" , Luigi Rizzo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 15:52:11 -0000 I think you meant "reply-all" :-) On Fri, Sep 27, 2013 at 7:53 AM, chintu hetam wrote: > As far as i know, flow director is Intel terminology it addresses both RSS > and RFS. I think FreeBSD implementation is RFS. > > Luigi, you touched upon SW de-multiplexer, i would like to know why it's > necessary. > let say i have 82599 ixgbe driver (RSS enabled)configured with 5 tuple > hash. My application reads from netmap queue 0-7(1-8), i know for sure that > each hash will be filtered to specific hw queue(0-7), is it safe to assume > netmap will provide packets in same order. > > Michio, reason i asked for performance values > http://arxiv.org/ftp/arxiv/papers/1106/1106.0443.pdf > I would like to test the accuracy of RFS,RSS and others in netmap mode.. > > Thanks > Hardik > > > On Fri, Sep 27, 2013 at 2:59 AM, hiren panchasara < > hiren.panchasara@gmail.com> wrote: > >> >> >> >> On Thu, Sep 26, 2013 at 2:27 PM, chintu hetam wrote: >> >>> Hiren, >>> >>> https://www.kernel.org/doc/Documentation/networking/scaling.txt must >>> read to understand nuances of each of this features. None of this >>> techniques are used for mostly none other than performance reasons. >>> >> >> Thanks for the link. >> So, RFS (Receive Flow Steering) is equivalent to "flow director" >> mentioned in FreeBSD's ixgbe drivers? >> >>> >>> Michio, personally i am interested to know performance results in netmap >>> mode with RFS patch you just mentioned. >>> >> Takuya/Luigi might have some numbers. >> >> Thanks, >> Hiren >> >> >> >> > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 16:49:26 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D86CDF70; Fri, 27 Sep 2013 16:49:26 +0000 (UTC) (envelope-from hiren.panchasara@gmail.com) Received: from mail-ea0-x229.google.com (mail-ea0-x229.google.com [IPv6:2a00:1450:4013:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 478AA2D3C; Fri, 27 Sep 2013 16:49:26 +0000 (UTC) Received: by mail-ea0-f169.google.com with SMTP id k11so1349407eaj.0 for ; Fri, 27 Sep 2013 09:49:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qtelpPZ2DyENQPw8Re9zo5NAlhP21Mzx44crNvUBJ0I=; b=tMpY379lUfExiVwDz8C8PGzNXanttVAAM7FeGK/eASk2Fqo1lDGf0OJfX7gMnVG8Np 5LbWQEZWVC2TazJwaUXug4hE5MSdyNpjfxD49bwHFT9TlNMmVZLXq6IcolOc8JErA9iu E7D33rmINJMvFOgfqq/io+MgEbI6BcjxNtqREQKNnHlEcV2Vx4CM3NWWfJIIyX/9UgCc 1DoLISlJ/R6w3dY8lGk6A4z2MDQ9jYGpYHfl0xEWHX5m8DiLUNp6FTWOqbpAVfkKPP0N 2xhoFbPOJHS9kZVpRM4yC9EEa8Bmxc3ykF/HU06W8RTJeXL5iV7awhrUGjxup+eaeBtV 72LQ== MIME-Version: 1.0 X-Received: by 10.15.45.8 with SMTP id a8mr11764556eew.1.1380300564612; Fri, 27 Sep 2013 09:49:24 -0700 (PDT) Received: by 10.14.105.137 with HTTP; Fri, 27 Sep 2013 09:49:24 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 09:49:24 -0700 Message-ID: Subject: Re: Adding Flow Director sysctls to ixgbe(4) (was: netmap: traffic distribution) From: hiren panchasara To: Takuya ASADA Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , Adrian Chadd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 16:49:26 -0000 On Fri, Sep 27, 2013 at 1:58 AM, Takuya ASADA wrote: > 2013/9/27 Adrian Chadd > >> On 27 September 2013 00:43, hiren panchasara wrote: >> >> >>> Takuya, >>> >>> I see a lot of responses/comments on proposed changes. Was anything >>> decided >>> at the end of it? As far as I can tell, its still not committed to the >>> tree. >>> >> >> I'd rather see an ioctl API for that chipset and then have a separate >> tool program it for now. >> > > Ah, like cxgbetool and cxgbe? (it has device specific tool and ioctls) > http://svnweb.freebsd.org/base/head/tools/tools/cxgbetool/ > Something like this for ixgbe would be nice to start with, imo. Cheers, Hiren > http://svnweb.freebsd.org/base/head/sys/dev/cxgb/ > > >> So, how bout we hack that up? :) >> > > Sound's interesting ;-) > Could you tell me more detail about your idea? > > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 18:16:27 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CCBD1FF0 for ; Fri, 27 Sep 2013 18:16:27 +0000 (UTC) (envelope-from rometoroam@gmail.com) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 58C5021F5 for ; Fri, 27 Sep 2013 18:16:27 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id hm2so1232958wib.10 for ; Fri, 27 Sep 2013 11:16:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZuKIrs/vcwmaFTFVTR87mutaSBtPzG5ZhwbVCmsGG6I=; b=p9ilzCvoko6tHZok7hPkU2xZqULBmJ0eFeMmSP2zVWDvTmKoV1UQlIDKPBEH6TUOVj RnyBn5jBKfXFmIB374zydcwS1HN9+I0FjyeHKHPylwOorFKNjDJaD+ZeiJ/lfB9T+Vgb MwPZOhAPMUX5ArhHyha/M8ZcSj4J/kTYcOZU/6wamidhr1iGsGjhw7PXXdzisiJPv3op ScEB3FRr3rH+oRZOVTdXjIHifujJEWF8kWx8HuYZidq2ybnm9NvW+SHskX4jwrTgdQbE HBiU2niQQchTk+P4RDJOXh7Oy2rRazv/sSj8fSs0SguIKbw1TQSMl4g+dNl8lpPyDSqJ sCtA== MIME-Version: 1.0 X-Received: by 10.180.12.4 with SMTP id u4mr3690117wib.29.1380305784971; Fri, 27 Sep 2013 11:16:24 -0700 (PDT) Received: by 10.194.64.167 with HTTP; Fri, 27 Sep 2013 11:16:24 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 14:16:24 -0400 Message-ID: Subject: Re: netmap: traffic distribution From: chintu hetam To: hiren panchasara Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Michio Honda , Luigi Rizzo , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 18:16:27 -0000 As far as i know, flow director is Intel terminology it addresses both RSS and RFS. I think FreeBSD implementation is RFS. Luigi, you touched upon SW de-multiplexer, i would like to know why it's necessary. let say i have 82599 ixgbe driver (RSS enabled)configured with 5 tuple hash. My application reads from netmap queue 0-7(1-8), for testing purpose i will define tuples in such a way that i know for sure that each packet will be filtered to specific hw queue(0-7). Now is it safe to assume that in netmap mode in user space i will receive the packet as in ixgbe hw queue? Michio, reason i asked for performance values http://arxiv.org/ftp/arxiv/papers/1106/1106.0443.pdf I would like to test the accuracy of RFS with HW RSS in netmap mode.. cheers, r2r On Fri, Sep 27, 2013 at 2:59 AM, hiren panchasara < hiren.panchasara@gmail.com> wrote: > > > > On Thu, Sep 26, 2013 at 2:27 PM, chintu hetam wrote: > >> Hiren, >> >> https://www.kernel.org/doc/Documentation/networking/scaling.txt must >> read to understand nuances of each of this features. None of this >> techniques are used for mostly none other than performance reasons. >> > > Thanks for the link. > So, RFS (Receive Flow Steering) is equivalent to "flow director" mentioned > in FreeBSD's ixgbe drivers? > >> >> Michio, personally i am interested to know performance results in netmap >> mode with RFS patch you just mentioned. >> > Takuya/Luigi might have some numbers. > > Thanks, > Hiren > > > > From owner-freebsd-net@FreeBSD.ORG Fri Sep 27 19:47:20 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 38883432 for ; Fri, 27 Sep 2013 19:47:20 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B3980275B for ; Fri, 27 Sep 2013 19:47:19 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id y6so2631379lbh.35 for ; Fri, 27 Sep 2013 12:47:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=DB9ElekQhlHN3bT+pq1pPVOBsU+c+LCtv/uxh+EKYHo=; b=rneTDh4nSJHfq+uLX7TBasyMHHZLEyaeo6bW7F1FmUbb5PphBy1ADm0bFIa18XFPL4 au3RecwLywXHuFX5wPQ9V5EBFMlJMfrCxDBBQ3V1pObiMDQUhNGlR0ivfnT7E+eQ0aFP Gx/sIxNJmi8Qyixk8waxT/EM/so9CZriKkC6LA59uW/IBjZnzHW/OwycAbiTIIGFfrXo lb15LFm0y/wgD/E1P8UUp35TtBwXWCdubglYQc/tbbCmlhfTJIZQSaYuzc5qCPkYm2zH nlVcAb+sOKocjywZ3M+SOvRxPMLxFw1oSaLPFuT3r42CdfxG8TlTwwal2bvnSBv7ICYg 5SHg== MIME-Version: 1.0 X-Received: by 10.152.228.130 with SMTP id si2mr3532574lac.32.1380311237530; Fri, 27 Sep 2013 12:47:17 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.172.105 with HTTP; Fri, 27 Sep 2013 12:47:17 -0700 (PDT) In-Reply-To: References: Date: Fri, 27 Sep 2013 21:47:17 +0200 X-Google-Sender-Auth: JPZrY_FzWKoEMUrLDRAFSpbEu1Q Message-ID: Subject: Re: netmap: traffic distribution From: Luigi Rizzo To: chintu hetam Content-Type: text/plain; charset=ISO-8859-1 Cc: Michio Honda , hiren panchasara , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2013 19:47:20 -0000 On Fri, Sep 27, 2013 at 8:16 PM, chintu hetam wrote: > As far as i know, flow director is Intel terminology it addresses both RSS > and RFS. I think FreeBSD implementation is RFS. > > Luigi, you touched upon SW de-multiplexer, i would like to know why it's > necessary. the sw demux is only necessary if you expect to have traffic with heavy processing requirements for which you cannot easily guarantee that the RSS more or less balances the queues. Since the number of RSS queues is small, unbalance is not unlikely; in software you could in principle demux each flow to its own queue and then have one worker process per thread process queues. Note though that scheduling may become complicated if you care for CPU locality, but if the sw demux only reads buffers, then the situation is not too bad. > let say i have 82599 ixgbe driver (RSS enabled)configured with 5 tuple hash. > My application reads from netmap queue 0-7(1-8), for testing purpose i will > define tuples in such a way that i know for sure that each packet will be > filtered to specific hw queue(0-7). Now is it safe to assume that in netmap > mode in user space i will receive the packet as in ixgbe hw queue? yes. > > Michio, reason i asked for performance values > http://arxiv.org/ftp/arxiv/papers/1106/1106.0443.pdf The problem that the paper describes is due to the incorrect implementation (at least according to the description give in the paper) of the Flow Director (FD) feature in linux. The hardware demux (RSS or RFS) per se does not cause reordering unless it is misued (as described in the paper). Basically when FD is active, the RFS entry in the NIC is generated by looking at the core used for the transmit side of a flow, and (according to what the paper says; not sure if it has been fixed since), the association is updated dynamically and not generated just once at the beginning of the flow. So if the userspace thread floats between CPUs, the RFS entry keepfloating, and incoming packets for the same flow can be delivered to different queues. cheers luigi -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2211611 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-net@FreeBSD.ORG Sat Sep 28 15:49:50 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8E1D6CEC for ; Sat, 28 Sep 2013 15:49:50 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5A13522BF for ; Sat, 28 Sep 2013 15:49:50 +0000 (UTC) Received: from sub-736ip109.rev.onenet.an ([190.88.36.109]:64760 helo=[10.1.17.227]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1VPwmA-0006u8-Qe; Sat, 28 Sep 2013 11:49:47 -0400 Content-Type: multipart/signed; boundary="Apple-Mail=_A45CAE38-AC98-44E2-9683-BD22C3A8BE5A"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Multiqueue support for bpf From: George Neville-Neil In-Reply-To: Date: Sat, 28 Sep 2013 11:49:44 -0400 Message-Id: References: To: Takuya ASADA X-Mailer: Apple Mail (2.1510) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: FreeBSD Net , Luigi Rizzo X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Sep 2013 15:49:50 -0000 --Apple-Mail=_A45CAE38-AC98-44E2-9683-BD22C3A8BE5A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Bump Has anyone else reviewed this code? I have looked it over but not run = it. Visually it looks fine to me. Best, George On Sep 4, 2013, at 4:04 , Takuya ASADA wrote: > Hi, >=20 > This is 2nd version of multiqueue bpf patch, I think I fixed things = what > you commented on previous mail. > Here's a change list of the patch: >=20 > - Drop added functions on struct > ifnet(if_get_[rt]xqueue_len/if_get_[rt]xqueue_affinity). > HW queue number and queue affinity informations are maybe useful for = some > applications, but it's not really directly related to multiqueue bpf. = I > think we should discuss them separately. >=20 > - Use BITSET for queue mask. > It seems better to use BITSET for queue mask structure, instead of = boolean > array. >=20 > - Drop tcpdump/libpcap changes. > It also should discuss separately. >=20 > - M_QUEUEID/IFCAP_QUEUEID > M_QUEUEID is the flag for mbuf which contains hw queue id. > IFCAP_QUEUEID is the flag which means the driver has ability to set = queue > id on mbuf. >=20 >=20 >=20 > 2013/7/3 Luigi Rizzo >=20 >>=20 >>=20 >>=20 >> On Tue, Jul 2, 2013 at 5:56 PM, Takuya ASADA = wrote: >>=20 >>> Hi, >>>=20 >>> Do you have an updated URL for the diffs ? The link below from your >>>> original message >>>> seems not working now (NXDOMAIN) >>>>=20 >>>> http://www.dokukino.com/mq_bpf_20110813.diff >>>>=20 >>>=20 >>> Changes with recent head is on my repository: >>> http://svnweb.freebsd.org/base/user/syuu/mq_bpf/ >>> And I attached a diff file on this mail. >>>=20 >>>=20 >> thanks for the diffs (the URL to the repo is useful too, >> but a URL to generate diffs is more convenient for reviewing = changes). >>=20 >> I believe it still needs a bit of work before being merged. >>=20 >> My comments (in order of the patch): >>=20 >> =3D=3D=3D ifnet.9 (and related code in if.c, sockio.h) =3D=3D=3D >> - if_get_rxqueue_len()/if_get_rxqueue_len() is not a good name, >> as to me at least it suggests that it returns the size of the >> individual queue, rather than the number of queues. >>=20 >> - cpu affinity (in userspace) is a bitmask, whereas in the BSD kernel >> we almost never use the term "affinity", and favour "couid" or = "oncpu" >> (i.e. an individual CPU id). >> I think you should either rename if_get_txqueue_affinity(), or make >> the return type a cpuset (which seems more sensible as the return >> value is passed to userspace) >>=20 >> =3D=3D=3D bpf.4 (and related code) =3D=3D=3D >>=20 >> - the ioctl() to attach/detach/report queues attached to a specific >> bpf descriptor talk about "mask bit" but neither the internal nor >> the external implementation deal with bits. >> I'd rather document those ioctl as "attaching queue to file = descriptor". >>=20 >> - the BPF ioctl names are generally inconsistent (using either S or = SET >> and G or GET for the setter and getter methods). >> But you should pick one of the patterns and stick with it, >> not introduce a third variant (GT/ST). >> Given we are in 2013 we might go for the long form GET and SET >> so i suggest the following (spaces for clarity) >>=20 >> +#define BIOC ENA QMASK _IO('B', 133) >> +#define BIOC DIS QMASK _IO('B', 134) >> +#define BIOC SET RXQMASK _IOWR('B', 135, uint32_t) >> +#define BIOC CLR RXQMASK _IOWR('B', 136, uint32_t) >> +#define BIOC GET RXQMASK _IOR('B', 137, uint32_t) >> +#define BIOC SET TXQMASK _IOWR('B', 138, uint32_t) >> +#define BIOC CLR TXQMASK _IOWR('B', 139, uint32_t) >> +#define BIOC GET TXQMASK _IOR('B', 140, uint32_t) >> +#define BIOC SET OTHERMASK _IO('B', 141) >> +#define BIOC CLR OTHERMASK _IO('B', 142) >> +#define BIOC GET OTHERMASK _IOR('B', 143, uint32_t) >>=20 >> Also related: the existing ioctls() use u_int as argumnts, rather >> than uint32_t. I personally prefer the uint32_t form, but you >> should at least add a comment to indicate that the choice is >> deliberate. >>=20 >> =3D=3D=3D if.c =3D=3D=3D >>=20 >>=20 >> - you have a KASSERT to report if ifp->if_get_*xqueue_affinity() is = not >> set, but i'd rather run the function only if is set, so you can >> have a multiqueue interface which does not bind queues to specific = cores >> (which i am not sure is always a great idea; too many processes >> statically bound to the same queue mean you lose opportunity to >> parallelize work.) >>=20 >> =3D=3D=3D mbuf.h =3D=3D=3D >>=20 >> as mentioned earlier, the modification to struct mbuf should >> be avoided if possible at all. It seems that you need just one >> direction bit (which maybe is available already from the context) >> and one queue identifier, which in the rx path, at least in your >> implementation is always a replica of the 'flowid' field. >> Can you see if perhaps the flowid field can be (ab)used on the >> tx path as well ? >>=20 >>=20 >> =3D=3D=3D if.h =3D=3D=3D >>=20 >> - in if.h, can you use individual variables instead of arrays >> for ifr_queue_affinity_index and friends ? >> The macros to map the fields of ifr_ifru one >> level up are a necessary evil, >> but there is no point in using the arrays. >>=20 >> - SIOCGIFTXQAFFINITY seems to use the receive function (copy&paste = typo) >> talks about >> Also, this function is probably something that should be coordinated >> with work on generic multiqueue support >>=20 >>=20 >> =3D=3D=3D bpf.c =3D=3D=3D >>=20 >> - in linux (and hopefully in FreeBSD at some point) the number of = queues >> can be changed at runtime. >> So i suggest that you cache the current number of queues when >> you allocate the arrays (qm_*xq_qmask[] ) rather than invoking >> ifp->if_get_*xqueue_len() everytime you need to do a boundary check. >> This will save us from all sort of problems later. >>=20 >> - in terms of code, the six BIOC*XQMASK are very similar, you are = probably >> better off having one single case in the switch >>=20 >> - can you some comments in the code for the chunk at @@ -2117,6 = +2391,42 @@ >> I do not completely understand why you are returning if the *queue = tag >> in the mbuf is out of range (my impression is that you should >> just continue, or if you think the packet is incorrect it should >> be filtered out before entering the LIST_FOREACH() ). >> Secondly, you should use the cached value of *queue_len >>=20 >>=20 >>=20 >> cheers >> luigi >>=20 >>=20 >> -- >> = -----------------------------------------+------------------------------- >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. = dell'Informazione >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> TEL +39-050-2211611 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >>=20 >> = -----------------------------------------+------------------------------- >>=20 >>=20 > = _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --Apple-Mail=_A45CAE38-AC98-44E2-9683-BD22C3A8BE5A Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEUEARECAAYFAlJG+pgACgkQYdh2wUQKM9KHGwCXTrt6bufqJtU9VaobGP7/7eWo 6gCePwVX7l60Wv9PLuDv/lgk99gSgjQ= =ddBo -----END PGP SIGNATURE----- --Apple-Mail=_A45CAE38-AC98-44E2-9683-BD22C3A8BE5A-- From owner-freebsd-net@FreeBSD.ORG Sat Sep 28 21:30:47 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C191AF63; Sat, 28 Sep 2013 21:30:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 60A9E23E8; Sat, 28 Sep 2013 21:30:47 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r8SLUcNX002065; Sun, 29 Sep 2013 00:30:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r8SLUcNX002065 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r8SLUcis002064; Sun, 29 Sep 2013 00:30:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 29 Sep 2013 00:30:38 +0300 From: Konstantin Belousov To: jfv@freebsd.org Subject: igb(4) panic: already enqueue Message-ID: <20130928213038.GT41229@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AhxFwN0LxQS9jVnS" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Sep 2013 21:30:47 -0000 --AhxFwN0LxQS9jVnS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On a relatively old test machine, dual-socket X7DWU supermicro, Xeon X5272, with on-board 82575EB igb controllers, running the stress2 udp blast test immediately causes the following panic: buf=3D0xfffff8001aa5c800 already enqueue at 1445 prod=3D1315 cons=3D= 1445 cpuid =3D 3 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff802839ab =3D db_trace_self_wrapper+0x2= b/frame 0xfffffe0123570600 vpanic() at 0xffffffff8033bd96 =3D vpanic+0x126/frame 0xfffffe0123570640 panic() at 0xffffffff8033be23 =3D panic+0x43/frame 0xfffffe01235706a0 igb_mq_start() at 0xffffffff80bcb228 =3D igb_mq_start+0xb8/frame 0xfffffe01= 23570710 =20 ether_output() at 0xffffffff804072a8 =3D ether_output+0x5c8/frame 0xfffffe0= 123570780 =20 ip_output() at 0xffffffff80430be1 =3D ip_output+0xe91/frame 0xfffffe0123570= 860 udp_send() at 0xffffffff804a5fc4 =3D udp_send+0x834/frame 0xfffffe0123570910 sosend_dgram() at 0xffffffff803b517b =3D sosend_dgram+0x36b/frame 0xfffffe0= 123570990 =20 soo_write() at 0xffffffff8039dbb9 =3D soo_write+0x49/frame 0xfffffe01235709= d0 dofilewrite() at 0xffffffff803956c5 =3D dofilewrite+0x85/frame 0xfffffe0123= 570a20 kern_writev() at 0xffffffff803953d5 =3D kern_writev+0x65/frame 0xfffffe0123= 570a70 sys_write() at 0xffffffff80395363 =3D sys_write+0x63/frame 0xfffffe0123570a= c0 amd64_syscall() at 0xffffffff8053ae7d =3D amd64_syscall+0x28d/frame 0xfffff= e0123570bf0 Test is available at svn:base/user/pho/stress2/testcases/udp, invocation line is BLASTHOST=3D192.168.5.1 ./udp -t 10m -i 50 -h -v Substitute the BLASTHOST with the ip address of a scratch box. This is on the yesterday HEAD. Could you, please, fix this ? --AhxFwN0LxQS9jVnS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.21 (FreeBSD) iQIcBAEBAgAGBQJSR0p9AAoJEJDCuSvBvK1B+agP/Au+5XtY7QsXwXVtFPpn1/uK doEc5bupoK6ROUQnudx3qSl/UERb6Jy0Bae31CvurirY+fw4LGUnhURNsydKxrtr 0jqlVBJiod9koKCe+P41Vlc1Md8vJyoSqKXJ/DApi2JAMqmihaXABDj3xEtLZfA9 V/jEeLn/sYw0hV9V5rmE+n04wlXI+wl8WPsllpho1mK8tspbPGkPJ7AcHqfm67kN +ZZLyP2dTYZIV12ibF47heGkRyayvb1E2NTDPsJdY+BqJFzDbB754HdAdXsCcQOv 6KG+1x7m1beYRmNwwAGVWnjScIRLTTHoHMO3HWUMuUfgvODgbg/FgyNHtjgPEQ3F nNo/ZPeaBUo34TFcnB00bziDYDLStM9VoPbPyBdB/tqFV+JOh7DjB7WAD1WJQFVT o4VlsG4E2aXO+hXKvRS/6d53HGGDKLiWdk30PvZhqMQwBtS91H1snpjda1DclQ3L UPgLTe6aiya32a5ZlFlXv56cqsDk+SQn7uiCO3gh8XNnpFQqRJtuj3v3Pm3EEU7y L0fkbGG3sGN7eQksPCPkbSv67Sxua27+AskDoSpGJ3fJSgOTr/FtALID6sEBusBx YyjXnX/8dklZrDlIik10PC53xEKKwfnRDEj66QDjiAktsYfUb8yQ5FZTN4LTK6iu s3zjI+yN/BuLbvI6TtGn =2v+W -----END PGP SIGNATURE----- --AhxFwN0LxQS9jVnS-- From owner-freebsd-net@FreeBSD.ORG Sat Sep 28 22:45:06 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D31BA983; Sat, 28 Sep 2013 22:45:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 71EB22722; Sat, 28 Sep 2013 22:45:06 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r8SMj16J017182; Sun, 29 Sep 2013 01:45:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r8SMj16J017182 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r8SMj1YZ017181; Sun, 29 Sep 2013 01:45:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 29 Sep 2013 01:45:01 +0300 From: Konstantin Belousov To: jfv@freebsd.org Subject: Re: igb(4) panic: already enqueue Message-ID: <20130928224501.GX41229@kib.kiev.ua> References: <20130928213038.GT41229@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XRLv71ZoqfThaM/0" Content-Disposition: inline In-Reply-To: <20130928213038.GT41229@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Sep 2013 22:45:07 -0000 --XRLv71ZoqfThaM/0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Sep 29, 2013 at 12:30:38AM +0300, Konstantin Belousov wrote: > On a relatively old test machine, dual-socket X7DWU supermicro, Xeon > X5272, with on-board 82575EB igb controllers, running the stress2 > udp blast test immediately causes the following >=20 > panic: buf=3D0xfffff8001aa5c800 already enqueue at 1445 prod=3D1315 cons= =3D1445 > cpuid =3D 3 > KDB: stack backtrace: > db_trace_self_wrapper() at 0xffffffff802839ab =3D db_trace_self_wrapper+0= x2b/frame 0xfffffe0123570600 > vpanic() at 0xffffffff8033bd96 =3D vpanic+0x126/frame 0xfffffe0123570640 > panic() at 0xffffffff8033be23 =3D panic+0x43/frame 0xfffffe01235706a0 > igb_mq_start() at 0xffffffff80bcb228 =3D igb_mq_start+0xb8/frame 0xfffffe= 0123570710 =20 > ether_output() at 0xffffffff804072a8 =3D ether_output+0x5c8/frame 0xfffff= e0123570780 =20 > ip_output() at 0xffffffff80430be1 =3D ip_output+0xe91/frame 0xfffffe01235= 70860 > udp_send() at 0xffffffff804a5fc4 =3D udp_send+0x834/frame 0xfffffe0123570= 910 > sosend_dgram() at 0xffffffff803b517b =3D sosend_dgram+0x36b/frame 0xfffff= e0123570990 =20 > soo_write() at 0xffffffff8039dbb9 =3D soo_write+0x49/frame 0xfffffe012357= 09d0 > dofilewrite() at 0xffffffff803956c5 =3D dofilewrite+0x85/frame 0xfffffe01= 23570a20 > kern_writev() at 0xffffffff803953d5 =3D kern_writev+0x65/frame 0xfffffe01= 23570a70 > sys_write() at 0xffffffff80395363 =3D sys_write+0x63/frame 0xfffffe012357= 0ac0 > amd64_syscall() at 0xffffffff8053ae7d =3D amd64_syscall+0x28d/frame 0xfff= ffe0123570bf0 >=20 > Test is available at > svn:base/user/pho/stress2/testcases/udp, invocation line is > BLASTHOST=3D192.168.5.1 ./udp -t 10m -i 50 -h -v > Substitute the BLASTHOST with the ip address of a scratch box. >=20 > This is on the yesterday HEAD. Could you, please, fix this ? I also tested on i350 adapters, there the issue is harder to reproduce, but it still panics under the test. Also, I saw the following: Expensive timeout(9) function: 0xffffffff809c7d60(0xfffffe0001864000) 1.929= 7931s Expensive timeout(9) function: 0xffffffff809c7d60(0xfffffe0001864000) 9.966= 9852s db> x/i 0xffffffff809c7d60 0xffffffff809c7d60 =3D igb_local_timer: pushq %rbp --XRLv71ZoqfThaM/0 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.21 (FreeBSD) iQIbBAEBAgAGBQJSR1vsAAoJEJDCuSvBvK1BTCUP+P79Xa3gbBhomSQUboyqW/hZ 1isDKCguScJo6hOW/R1iltnkbn3RMCJNBQS9MBM9dWdiVGI0orBolsk/XKKKCT8l W2pPWjlInN0jUPqoBDj+flF1eM9d0EDnuJVMKIlESMD5cuY+bpMm+OLZo6u8qqwR 6O1Db2kUt4kgqhbjyrjTj65Isdrll0Pa5vSMGMQe1/Y9UI3aBJQVxmJdsgBE8IbZ h3x4Vus6fgzBHjikrYQGer8pUCnc32BfpsJfNiiyKB5ki3ix6mXn6BfAj197r1H+ N++nJ4g6UvHl9YfNoSxRae981FDuqwYod1PfYm1RkMQh9UaB6l+hkihhDI+ncqdS 8NjC4tDaF+NGtuKFc/sX6PTxbrJ4Y5uxUU6Rc3xdcW9wb+7ZHwnDfh/kBumtzh2w sFEVsclx5XgCZfU6PjHLhHs3KawB37jRiiS/5k/98P9ohkOQvRQyuAZMqsghkvyL upRrFl+MtA7koyoZBMvGPqvdA3/ROXW2njUiQitlLPuj/s6KJVUKF9uqGL+hbVjW F2Y9mMM1N7BfsiuEkW0pDXENWquBiu4LcKGo+c3WnsJH5JDkxPblSkYiwvg2LAdj ozf2GmP+6Ug5DEsDfZpMm6PDp5TqPI8rfLmMU6h+FjfCe8TqGQt76x3gCBoAtixt z7gMtMRV1XhQEqkXu44= =c29f -----END PGP SIGNATURE----- --XRLv71ZoqfThaM/0--