Date: Wed, 2 Oct 2013 16:15:20 -0700 From: hiren panchasara <hiren.panchasara@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: netmap: understanding pkg-gen.c Message-ID: <CALCpEUFA6Z3ASgKFKPsWjt1HfUCFNM1iAhSte5ChNHLpi6oJLw@mail.gmail.com> In-Reply-To: <CALCpEUGi%2Boaq9JftSWX%2BR-UL%2BCVHaYwpmjvzksKANjhLcn-niA@mail.gmail.com> References: <CALCpEUHj_Mke_njM0NwzCUa1f_pXZdCy3_aaqJtKz5NUJUU8Mg@mail.gmail.com> <CA%2BhQ2%2BjYyoxpLEoz0kHm8Hikfgf8Fke-o_ctkO3fKBdGqEz7wQ@mail.gmail.com> <CALCpEUG1dM9Z6OG1BqRAPAfHciANXxaYVpezDEOUNug22QN6HQ@mail.gmail.com> <CALCpEUGi%2Boaq9JftSWX%2BR-UL%2BCVHaYwpmjvzksKANjhLcn-niA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 2, 2013 at 3:11 PM, hiren panchasara <hiren.panchasara@gmail.com
> wrote:
>
>
>
> On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara <
> hiren.panchasara@gmail.com> wrote:
>
>>
>> Thanks Luigi.
>>
>> Coming back to this thread to actually understand what's going on.
>>
>>
>> On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
>>
>>> >
>>> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1
>>> (10.73.149.17)
>>> > and this is how I am using this binary:
>>> >
>>> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
>>> > 10.73.149.17 -s 10.73.149.28
>>>
>>
>> So, my intention is to *send* 10gbps data to ix1 and see the card use all
>> of its 8 queues.
>>
>> Above command is the correct one?
>>
>> I kldunloaded/loaded ixbge to clear out all the stats and tested it again:
>>
>>
>> -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
>> 10.73.149.17 -s 10.73.149.28
>> extract_ip_range [143] extract IP range from 10.73.149.28
>> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
>> extract_ip_range [143] extract IP range from 10.73.149.17
>> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
>> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
>> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
>> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
>> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
>> main [1530] map size is 334980 Kb
>> main [1552] mapping 334980 Kbytes
>> Sending on ix1: 8 queues, 8 threads and 8 cpus.
>> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
>> main [1622] Sending 512 packets every 0.000000000 ns
>> main [1624] Wait 2 secs for phy reset
>> main [1626] Ready...
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [775] start
>> sender_body [848] drop copy
>>
>> sender_body [775] start
>> sender_body [848] drop copy
>> sender_body [848] drop copy
>> sender_body [848] drop copy
>> sender_body [848] drop copy
>> sender_body [848] drop copy
>> sender_body [848] drop copy
>> main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec)
>> main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec)
>> sender_body [841] poll error/timeout on queue 1
>> main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec)
>> main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec)
>> main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec)
>> main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec)
>> main_thread [1212] ouch, thread 1 exited with error
>> Sent 87502047 packets, 60 bytes each, in 5.86 seconds.
>> Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps)
>> -bash-4.2$
>>
>> But looking at the queue stats:
>>
>>
>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
>> dev.ix.1.queue0.rx_packets: 171
>> dev.ix.1.queue1.rx_packets: 0
>> dev.ix.1.queue2.rx_packets: 0
>> dev.ix.1.queue3.rx_packets: 0
>> dev.ix.1.queue4.rx_packets: 0
>> dev.ix.1.queue5.rx_packets: 0
>> dev.ix.1.queue6.rx_packets: 0
>> dev.ix.1.queue7.rx_packets: 0
>>
>> And after a few seconds:
>>
>> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
>> dev.ix.1.queue0.rx_packets: 310
>> dev.ix.1.queue1.rx_packets: 0
>> dev.ix.1.queue2.rx_packets: 0
>> dev.ix.1.queue3.rx_packets: 0
>> dev.ix.1.queue4.rx_packets: 8
>> dev.ix.1.queue5.rx_packets: 0
>> dev.ix.1.queue6.rx_packets: 0
>> dev.ix.1.queue7.rx_packets: 0
>> -bash-4.2$
>>
>> What is going on here? Should I be seeing more pkts in rx_packets? Should
>> I see more queues being used?
>>
>> I am using stock ixgbe at this point. I believe RSS is enabled by
>> default?
>>
>
> fdir(flow director)/atr is also enabled by default.
>
> I tried to turn that off by following change in /sys/modules/ixgbe/Makefile
>
> -CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR
> +CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP
>
> But, now interface would not attach to netmap:
>
>
> sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s
> 10.73.149.28
> extract_ip_range [143] extract IP range from 10.73.149.28
> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
> extract_ip_range [143] extract IP range from 10.73.149.17
> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
> main [1530] map size is 334980 Kb
> main [1536] Unable to get if info for ix1
> main [1543] bad nthreads 8, have 0 queues
> main [1552] mapping 0 Kbytes
> main [1558] Unable to mmap 0 KB
> main [1572] Unable to register interface ix1
>
> Sending on ix1: 0 queues, 8 threads and 8 cpus.
> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
> main [1622] Sending 512 packets every 0.000000000 ns
> main [1624] Wait 2 secs for phy reset
> main [1626] Ready...
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> start_threads [1126] Unable to register ix1
> main_thread [1192] 0 pps (0 pkts in 1043812 usec)
> main_thread [1212] ouch, thread 0 exited with error
>
> main_thread [1212] ouch, thread 1 exited with error
> main_thread [1212] ouch, thread 2 exited with error
> main_thread [1212] ouch, thread 3 exited with error
> main_thread [1212] ouch, thread 4 exited with error
> main_thread [1212] ouch, thread 5 exited with error
> main_thread [1212] ouch, thread 6 exited with error
> main_thread [1212] ouch, thread 7 exited with error
> Sent 0 packets, 60 bytes each, in 0.00 seconds.
> Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps)
>
> Error is coming from pkg-gen.c:
>
> if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) {
> D("Unable to register %s", g->ifname);
> continue;
> }
>
So somehow I am always seeing this error now. No matter what changes I have
made to ixgbe. I've tried a reboot too.
I've removed "device ixgbe" from kernconf and building it as a module.
Should I not do that?
Let me try putting it back in the kernel and see if I am still seeing the
issue.
Really weird.
cheers,
Hiren
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALCpEUFA6Z3ASgKFKPsWjt1HfUCFNM1iAhSte5ChNHLpi6oJLw>