From owner-soc-status@freebsd.org Sun Aug 30 00:09:21 2015 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE7B19C5999 for ; Sun, 30 Aug 2015 00:09:21 +0000 (UTC) (envelope-from btw@mail.ustc.edu.cn) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id 8D8181889; Sun, 30 Aug 2015 00:09:20 +0000 (UTC) (envelope-from btw@mail.ustc.edu.cn) Received: from dell (unknown [182.112.30.224]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygAH6sKlSeJVNROgAA--.6977S2; Sun, 30 Aug 2015 08:09:15 +0800 (CST) Date: Sun, 30 Aug 2015 08:09:34 +0800 From: Tiwei Bie To: George Neville-Neil Cc: soc-status@freebsd.org, hiren@FreeBSD.org, rwatson@FreeBSD.org Subject: Re: Multiqueue testing project - final status report Message-ID: <20150830000429.GA11226@dell> References: <1440669563-38392-1-git-send-email-btw@mail.ustc.edu.cn> <3C4FC610-7052-4580-95CA-731097A15F92@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <3C4FC610-7052-4580-95CA-731097A15F92@freebsd.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygAH6sKlSeJVNROgAA--.6977S2 X-Coremail-Antispam: 1UD129KBjvJXoWxGrWUtr4kWrW3Cr15urWfKrg_yoW7Gr4xpF yay3W7tr4kJF13Xa4xtw18Xr1Ivrs8GrWrWr1fKry8u3Z8uF1rtr4SvFW0yayUXFW0qay2 vw4DXr4kAr4DAaDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUyab7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW8Jr0_Cr1UM28EF7xvwV C2z280aVCY1x0267AKxVW8Jr0_Cr1UM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVAC Y4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJV W8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41l42xK82IYc2Ij64vIr41l4I8I 3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxV WUGVWUWwC2zVAF1VAY17CE14v26r126r1DMIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAF wI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r1j6r4UMIIF0xvE42xK8VAvwI8IcI k0rVWrZr1j6s0DMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_ Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU5PpnJUUUUU== X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQYQAVQhmDZe8AABsi X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Aug 2015 00:09:22 -0000 On Sat, Aug 29, 2015 at 08:54:10PM +0100, George Neville-Neil wrote: > > > On 27 Aug 2015, at 10:59, Tiwei Bie wrote: > > > Dear All! > > > > This is the final status report of the Multiqueue Testing Project. > > > > The aim of this project is to design and implement an infrastructure > > to validate that a number of the network stack's multiqueue behaviours > > are as expected. > > > > The deliverables of this project mainly consist of: > > > > - A general mechanism to collect the per-ring per-cpu statistics > > which can be used by all NIC drivers, and the extended netstat(1) > > which can report these statistics. > > > > - A suite of network stack behavior testing programs which consists > > of (a) a virtual multiqueue ethernet interface (vme), (b) a UDP > > packet generator based on vme, (c) a UDP server based on socket(2), > > (d) a TCP client based on lwip and vme, (e) a TCP server based on > > socket. > > > > At present, most parts of this project have been implemented. Few > > parts have been committed to -HEAD, and other parts have been submitted > > to Phabricator. > > > > Now, I'm also working on improving the IPv6 RSS supports with the > > help of adrian@. ^_^ > > > > Finally, I want to say thank you to my mentor Hiren, George and Robert, > > for the helps and guidance! ^_^ > > > > Howdy, > > I've seen quite a few commits go by which is most excellent. Quick question, > where are the tests? Did I miss those? > I only sent the test to Adrian when replying his mail. I should submit the test to Phabricator together with the patch... Sorry... :-( PS. This is the test I sent to Adrian: > On Wed, Aug 26, 2015 at 10:38:30PM -0700, Adrian Chadd wrote: > > ooo cool! ok, so have you tested it at all? :) If so, how'd you test it? > > > > I have tested it with the tools in tools/tools/mq-testing [1]. ^_^ > I set each packet's hash type to M_HASHTYPE_NONE for the packets > injected by pktgen [2] via vme [3]. So the packets are received with > no RSS info setted. And it will be done a software calculation of > the RSS by rss_soft_m2cpuid_v6(). > > The command I used to inject packets to network stack via vme: > > % sudo ./pktgen -i vme0 -6 > > vme is configured to have 4 queues: > > % sysctl net.link.vme.num_queues > net.link.vme.num_queues: 4 > > I have 4 cores on my computer, And netisr is configured to use 4 > threads: > > % cat /boot/loader.conf > net.isr.maxthreads=4 > net.isr.bindthreads=1 > > And I got the following outputs from netstat [4]: > > % ./netstat -I vme0 -R > --------------------------- vme0 ------------------------ > > ring0: > cpu0 cpu1 cpu2 cpu3 > ifinput : 39648737 37519099 37491641 37382462 > netisr : 112393348 0 0 0 > ether : 39648732 37519102 37491643 37382462 > ip : 0 0 0 0 > ip6 : 152041939 0 0 0 > tcp : 0 0 0 0 > udp : 151189032 0 0 0 > > ring1: > cpu0 cpu1 cpu2 cpu3 > ifinput : 39639711 37518364 37496728 37383776 > netisr : 0 114520209 0 0 > ether : 39639710 37518363 37496724 37383782 > ip : 0 0 0 0 > ip6 : 0 152038579 0 0 > tcp : 0 0 0 0 > udp : 0 151183691 0 0 > > ring2: > cpu0 cpu1 cpu2 cpu3 > ifinput : 39643992 37520994 37503514 37382789 > netisr : 0 0 114547782 0 > ether : 39643999 37520995 37503505 37382790 > ip : 0 0 0 0 > ip6 : 0 0 152051289 0 > tcp : 0 0 0 0 > udp : 0 0 151195834 0 > > ring3: > cpu0 cpu1 cpu2 cpu3 > ifinput : 39650698 37524393 37504604 37382778 > netisr : 0 0 0 114679712 > ether : 39650694 37524394 37504604 37382781 > ip : 0 0 0 0 > ip6 : 0 0 0 152062473 > tcp : 0 0 0 0 > udp : 0 0 0 151206219 > > This table shows the number of packets received on each ring > and processed by each layer on each CPU. So the following > line means: > > ring0: > cpu0 cpu1 cpu2 cpu3 > ifinput : 39648737 37519099 37491641 37382462 > > There are 39648737 packets received on ring0 are processed by > ifp->if_input(), ie. ether_input() on CPU0. > > Because the pktgen wasn't bind to a specified CPU, and ifp->if_input(), > ie. ether_input() is executed on pktgen's process context. So > the 'ifinput' row shows the packets have been processed by ether_input() > on each CPU. > > As "ether" netisr handler's dispatch is NETISR_DISPATCH_DIRECT, > the packets are processed by ether_nh_input() on the same CPU. > > As "ip6" netisr handler's dispatch is NETISR_DISPATCH_HYBRID, > a new CPU will be selected by rss_soft_m2cpuid_v6(). So the > packets are processed by ip6_input() on a selected CPU based > on the RSS hashing. > > The packets injected to vme are assigned to a selected ring > based on the RSS hashing. So the packets received on each ring > are processed by ip6/udp on a corresponding CPU. It is the > same with what the table shows. So rss_soft_m2cpuid_v6() works. > > Sorry for my broken English... :-( > > Best regards, > Twei Bie > > [1] https://svnweb.freebsd.org/socsvn/soc2015/btw/head/tools/tools/mq-testing > [2] https://svnweb.freebsd.org/socsvn/soc2015/btw/head/tools/tools/mq-testing/udp > [3] https://svnweb.freebsd.org/socsvn/soc2015/btw/head/tools/tools/mq-testing/vme > [4] https://svnweb.freebsd.org/socsvn/soc2015/btw/head/usr.bin/netstat Best regards, Tiwei Bie