Date: Wed, 2 Oct 2013 15:11:07 -0700 From: hiren panchasara <hiren.panchasara@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: netmap: understanding pkg-gen.c Message-ID: <CALCpEUGi%2Boaq9JftSWX%2BR-UL%2BCVHaYwpmjvzksKANjhLcn-niA@mail.gmail.com> In-Reply-To: <CALCpEUG1dM9Z6OG1BqRAPAfHciANXxaYVpezDEOUNug22QN6HQ@mail.gmail.com> References: <CALCpEUHj_Mke_njM0NwzCUa1f_pXZdCy3_aaqJtKz5NUJUU8Mg@mail.gmail.com> <CA%2BhQ2%2BjYyoxpLEoz0kHm8Hikfgf8Fke-o_ctkO3fKBdGqEz7wQ@mail.gmail.com> <CALCpEUG1dM9Z6OG1BqRAPAfHciANXxaYVpezDEOUNug22QN6HQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 2, 2013 at 1:18 PM, hiren panchasara <hiren.panchasara@gmail.com
> wrote:
>
> Thanks Luigi.
>
> Coming back to this thread to actually understand what's going on.
>
>
> On Tue, Sep 24, 2013 at 8:37 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
>
>> >
>> > On this box, I have 2 interfaces igb0 (10.73.149.28) and ix1
>> (10.73.149.17)
>> > and this is how I am using this binary:
>> >
>> > -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
>> > 10.73.149.17 -s 10.73.149.28
>>
>
> So, my intention is to *send* 10gbps data to ix1 and see the card use all
> of its 8 queues.
>
> Above command is the correct one?
>
> I kldunloaded/loaded ixbge to clear out all the stats and tested it again:
>
>
> -bash-4.2$ sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d
> 10.73.149.17 -s 10.73.149.28
> extract_ip_range [143] extract IP range from 10.73.149.28
> extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
> extract_ip_range [143] extract IP range from 10.73.149.17
> extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
> extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
> extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
> extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
> extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
> main [1530] map size is 334980 Kb
> main [1552] mapping 334980 Kbytes
> Sending on ix1: 8 queues, 8 threads and 8 cpus.
> 10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
> main [1622] Sending 512 packets every 0.000000000 ns
> main [1624] Wait 2 secs for phy reset
> main [1626] Ready...
> sender_body [775] start
> sender_body [775] start
> sender_body [775] start
> sender_body [775] start
> sender_body [775] start
> sender_body [775] start
> sender_body [775] start
> sender_body [848] drop copy
>
> sender_body [775] start
> sender_body [848] drop copy
> sender_body [848] drop copy
> sender_body [848] drop copy
> sender_body [848] drop copy
> sender_body [848] drop copy
> sender_body [848] drop copy
> main_thread [1192] 15122963 pps (15130434 pkts in 1000494 usec)
> main_thread [1192] 14881444 pps (14896266 pkts in 1000996 usec)
> sender_body [841] poll error/timeout on queue 1
> main_thread [1192] 14880708 pps (15659371 pkts in 1052327 usec)
> main_thread [1192] 14878611 pps (14888684 pkts in 1000677 usec)
> main_thread [1192] 14882655 pps (14897538 pkts in 1001000 usec)
> main_thread [1192] 11900044 pps (12029754 pkts in 1010900 usec)
> main_thread [1212] ouch, thread 1 exited with error
> Sent 87502047 packets, 60 bytes each, in 5.86 seconds.
> Speed: 14.92 Mpps Bandwidth: 7.16 Gbps (raw 10.03 Gbps)
> -bash-4.2$
>
> But looking at the queue stats:
>
>
> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
> dev.ix.1.queue0.rx_packets: 171
> dev.ix.1.queue1.rx_packets: 0
> dev.ix.1.queue2.rx_packets: 0
> dev.ix.1.queue3.rx_packets: 0
> dev.ix.1.queue4.rx_packets: 0
> dev.ix.1.queue5.rx_packets: 0
> dev.ix.1.queue6.rx_packets: 0
> dev.ix.1.queue7.rx_packets: 0
>
> And after a few seconds:
>
> -bash-4.2$ sysctl -a | grep ix.1 | grep queue | grep rx_packets
> dev.ix.1.queue0.rx_packets: 310
> dev.ix.1.queue1.rx_packets: 0
> dev.ix.1.queue2.rx_packets: 0
> dev.ix.1.queue3.rx_packets: 0
> dev.ix.1.queue4.rx_packets: 8
> dev.ix.1.queue5.rx_packets: 0
> dev.ix.1.queue6.rx_packets: 0
> dev.ix.1.queue7.rx_packets: 0
> -bash-4.2$
>
> What is going on here? Should I be seeing more pkts in rx_packets? Should
> I see more queues being used?
>
> I am using stock ixgbe at this point. I believe RSS is enabled by default?
>
fdir(flow director)/atr is also enabled by default.
I tried to turn that off by following change in /sys/modules/ixgbe/Makefile
-CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP -DIXGBE_FDIR
+CFLAGS+= -I${.CURDIR}/../../dev/ixgbe -DSMP
But, now interface would not attach to netmap:
sudo ./pkt-gen -i ix1 -f tx -n 100000000 -c 8 -p 8 -d 10.73.149.17 -s
10.73.149.28
extract_ip_range [143] extract IP range from 10.73.149.28
extract_ip_range [178] range is 10.73.149.28 0 to 10.73.149.28 0
extract_ip_range [143] extract IP range from 10.73.149.17
extract_ip_range [178] range is 10.73.149.17 0 to 10.73.149.17 0
extract_mac_range [184] extract MAC range from 90:e2:ba:30:68:c5
extract_mac_range [199] 90:e2:ba:30:68:c5 starts at 90:e2:ba:30:68:c5
extract_mac_range [184] extract MAC range from ff:ff:ff:ff:ff:ff
extract_mac_range [199] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff
main [1530] map size is 334980 Kb
main [1536] Unable to get if info for ix1
main [1543] bad nthreads 8, have 0 queues
main [1552] mapping 0 Kbytes
main [1558] Unable to mmap 0 KB
main [1572] Unable to register interface ix1
Sending on ix1: 0 queues, 8 threads and 8 cpus.
10.73.149.28 -> 10.73.149.17 (90:e2:ba:30:68:c5 -> ff:ff:ff:ff:ff:ff)
main [1622] Sending 512 packets every 0.000000000 ns
main [1624] Wait 2 secs for phy reset
main [1626] Ready...
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
start_threads [1126] Unable to register ix1
main_thread [1192] 0 pps (0 pkts in 1043812 usec)
main_thread [1212] ouch, thread 0 exited with error
main_thread [1212] ouch, thread 1 exited with error
main_thread [1212] ouch, thread 2 exited with error
main_thread [1212] ouch, thread 3 exited with error
main_thread [1212] ouch, thread 4 exited with error
main_thread [1212] ouch, thread 5 exited with error
main_thread [1212] ouch, thread 6 exited with error
main_thread [1212] ouch, thread 7 exited with error
Sent 0 packets, 60 bytes each, in 0.00 seconds.
Speed: 0.00 pps Bandwidth: 0.00 bps (raw 0.00 bps)
Error is coming from pkg-gen.c:
if ((ioctl(tfd, NIOCREGIF, &tifreq)) == -1) {
D("Unable to register %s", g->ifname);
continue;
}
Cheers,
Hiren
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALCpEUGi%2Boaq9JftSWX%2BR-UL%2BCVHaYwpmjvzksKANjhLcn-niA>