Date: Fri, 22 Mar 2013 12:46:14 +0100 From: Walter de Donato <walter.dedonato@unina.it> To: Michio Honda <micchie@sfc.wide.ad.jp> Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: Netmap ixgbe driver problem on 10Gbps X540-AT2 adapters (Linux only?) Message-ID: <CAPRxaNSZBe0zPybwwyS4AtOhQ1Z1U5Ly4ZPDhAtFm7s%2BP84zgw@mail.gmail.com> In-Reply-To: <D9E78ACE-2E40-4EE8-AC0F-2BCA8DA6D158@sfc.wide.ad.jp> References: <CAPRxaNQVkkb1nX-4ziL6BYp_pLqNAvZruYUEPQs_4=Xvinc2gg@mail.gmail.com> <D9E78ACE-2E40-4EE8-AC0F-2BCA8DA6D158@sfc.wide.ad.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you Michio, actually for me it was necessary to set -w 6 to avoid the adapter reset every second, but what I am observing is a big difference between what is being sent and what is being received on the other side, as reported in the following. This experiment relates to two separate Linux servers connected back to back with a CAT6 cable. Sender output: $ sudo ./pkt-gen -i eth2 -n 100000000 -w 6 -f tx extract_ip_range [136] extract IP range from 10.0.0.1 extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 extract_ip_range [136] extract IP range from 10.1.0.1 extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 extract_mac_range [177] extract MAC range from a0:36:9f:0d:f8:88 extract_mac_range [192] a0:36:9f:0d:f8:88 starts at a0:36:9f:d:f8:88 extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff main [1401] map size is 334980 Kb main [1423] mapping 334980 Kbytes Sending on eth2: 16 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f8:88 -> ff:ff:ff:ff:ff:ff) main [1477] Wait 6 secs for phy reset main [1479] Ready... sender_body [683] start sender_body [736] drop copy main_thread [1078] 8722463 pps (8731028 pkts in 1000982 usec) main_thread [1078] 8416235 pps (8425308 pkts in 1001078 usec) main_thread [1078] 8498540 pps (8506792 pkts in 1000971 usec) main_thread [1078] 9048765 pps (9058755 pkts in 1001104 usec) main_thread [1078] 9136045 pps (9138740 pkts in 1000295 usec) main_thread [1078] 9062717 pps (9069541 pkts in 1000753 usec) main_thread [1078] 8693680 pps (8703017 pkts in 1001074 usec) main_thread [1078] 8920226 pps (8930172 pkts in 1001115 usec) main_thread [1078] 8879883 pps (8882067 pkts in 1000246 usec) main_thread [1078] 9060322 pps (9070343 pkts in 1001106 usec) main_thread [1078] 9086352 pps (9096556 pkts in 1001123 usec) main_thread [1078] 2384776 pps (2387681 pkts in 1001218 usec) Sent 100000000 packets, 60 bytes each, in 11.27 seconds. Speed: 8.87 Mpps Bandwidth: 4.26 Gbps (raw 5.96 Gbps) Receiver output: $ sudo ./pkt-gen -f rx -i eth2 -w 6 extract_ip_range [136] extract IP range from 10.0.0.1 extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 extract_ip_range [136] extract IP range from 10.1.0.1 extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98 extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98 extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff main [1401] map size is 334980 Kb main [1423] mapping 334980 Kbytes Receiving from eth2: 4 queues, 1 threads and 1 cpus. main [1477] Wait 6 secs for phy reset main [1479] Ready... main_thread [1078] 0 pps (0 pkts in 1001034 usec) receiver_body [833] waiting for initial packets, poll returns 0 0 main_thread [1078] 0 pps (0 pkts in 1001028 usec) receiver_body [833] waiting for initial packets, poll returns 0 0 main_thread [1078] 322409 pps (324924 pkts in 1007802 usec) main_thread [1078] 339256 pps (340996 pkts in 1005129 usec) main_thread [1078] 339305 pps (345252 pkts in 1017528 usec) main_thread [1078] 339285 pps (341276 pkts in 1005868 usec) main_thread [1078] 339284 pps (340752 pkts in 1004326 usec) main_thread [1078] 339271 pps (341316 pkts in 1006028 usec) main_thread [1078] 339276 pps (344396 pkts in 1015090 usec) main_thread [1078] 339271 pps (341420 pkts in 1006335 usec) main_thread [1078] 339293 pps (340960 pkts in 1004914 usec) main_thread [1078] 339291 pps (341528 pkts in 1006594 usec) main_thread [1078] 339270 pps (344080 pkts in 1014177 usec) main_thread [1078] 185293 pps (185491 pkts in 1001068 usec) main_thread [1078] 0 pps (0 pkts in 1001085 usec) Received 3932391 packets, in 11.59 seconds. Speed: 339.29 Kpps -Walter 2013/3/22 Michio Honda <micchie@sfc.wide.ad.jp> > You need to specify "-w 4" on running pkt-gen to wait for that the link > resets. > > Cheers, > - Michio > > On Mar 22, 2013, at 11:16 AM, Walter de Donato wrote: > > > Dear Tahir, > > > > I've already tried to disable Rx/Tx pause and autonegotiation but the > > result in transmission is always the same. > > > > I did the following: > > $ sudo ifconfig eth2 up > > $ sudo ethtool -A eth2 autoneg off > > $ sudo ethtool -A eth2 rx off > > $ sudo ethtool -A eth2 tx off > > > > These are the resulting dmesg output lines: > > [253131.445124] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow > > Control: RX/TX > > [253147.935426] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow > > Control: TX > > [253163.930776] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow > > Control: None > > > > Here is what I obtain: > > $ sudo ./pkt-gen -i eth2 -n 100000 -f tx > > extract_ip_range [136] extract IP range from 10.0.0.1 > > extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 > > extract_ip_range [136] extract IP range from 10.1.0.1 > > extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 > > extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98 > > extract_mac_range [192] a0:36:9f:0d:f0:98 starts at a0:36:9f:d:f0:98 > > extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff > > extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at ff:ff:ff:ff:ff:ff > > main [1401] map size is 334980 Kb > > main [1423] mapping 334980 Kbytes > > Sending on eth2: 4 queues, 1 threads and 1 cpus. > > 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff) > > main [1477] Wait 2 secs for phy reset > > main [1479] Ready... > > sender_body [683] start > > main_thread [1078] 7155 pps (7162 pkts in 1001020 usec) > > main_thread [1078] 3484 pps (3488 pkts in 1001030 usec) > > main_thread [1078] 3487 pps (3491 pkts in 1001024 usec) > > main_thread [1078] 3466 pps (3469 pkts in 1000847 usec) > > main_thread [1078] 3485 pps (3490 pkts in 1001521 usec) > > main_thread [1078] 3322 pps (3328 pkts in 1001765 usec) > > main_thread [1078] 3293 pps (3297 pkts in 1001322 usec) > > main_thread [1078] 3492 pps (3498 pkts in 1001763 usec) > > main_thread [1078] 3485 pps (3489 pkts in 1001168 usec) > > main_thread [1078] 3351 pps (3356 pkts in 1001402 usec) > > main_thread [1078] 3297 pps (3301 pkts in 1001314 usec) > > main_thread [1078] 3933 pps (3938 pkts in 1001390 usec) > > main_thread [1078] 3481 pps (3486 pkts in 1001528 usec) > > main_thread [1078] 3493 pps (3497 pkts in 1001177 usec) > > main_thread [1078] 3310 pps (3314 pkts in 1001292 usec) > > main_thread [1078] 3485 pps (3491 pkts in 1001812 usec) > > main_thread [1078] 3326 pps (3331 pkts in 1001547 usec) > > main_thread [1078] 3480 pps (3485 pkts in 1001456 usec) > > main_thread [1078] 3489 pps (3493 pkts in 1001209 usec) > > main_thread [1078] 3377 pps (3381 pkts in 1001040 usec) > > main_thread [1078] 3450 pps (3454 pkts in 1001048 usec) > > main_thread [1078] 3378 pps (3382 pkts in 1001050 usec) > > main_thread [1078] 3470 pps (3474 pkts in 1001052 usec) > > main_thread [1078] 3323 pps (3327 pkts in 1001056 usec) > > main_thread [1078] 3482 pps (3486 pkts in 1001050 usec) > > main_thread [1078] 3472 pps (3476 pkts in 1001066 usec) > > main_thread [1078] 3497 pps (3501 pkts in 1001044 usec) > > main_thread [1078] 3115 pps (3115 pkts in 1000035 usec) > > main_thread [1078] 0 pps (0 pkts in 1001057 usec) > > Sent 100000 packets, 60 bytes each, in 28.42 seconds. > > Speed: 3.52 Kpps Bandwidth: 1.69 Mbps (raw 2.36 Mbps) > > > > While executing pkt-gen I see the following dmesg output lines: > > [253741.892031] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253742.874686] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253743.864635] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253744.854513] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253745.844527] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253746.830323] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253747.821223] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253748.813229] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253749.804317] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253750.794286] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253751.784302] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253752.774324] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253753.764258] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253754.753735] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253755.748011] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253756.734927] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253757.724876] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253758.713267] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253759.704793] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253760.694759] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253761.684743] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253762.674699] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253763.664642] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253764.654488] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253765.644462] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253766.634425] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253767.624392] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253768.614364] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253769.611107] ixgbe 0000:0c:00.0: eth2: Reset adapter > > [253769.715298] 893.960705 netmap_ring_reinit [966] called for eth2 > > [253769.716646] 893.962054 netmap_ring_reinit [966] called for eth2 > > [253769.725844] 893.971252 netmap_ioctl [1236] deprecated, data is > ded4df20 > > [253776.141936] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 Gbps, Flow > > Control: None > > > > It seems that the adapter is reset every second, but no packet goes on > the > > wire. > > > > I'm starting to look into the code to investigate this issue. > > If you have any suggestion on where to look at it would save me a lot of > > time. > > > > I also rechecked the rx mode. > > It is working at least up to 50Kpps (as I'm generation on the other side > > with nping and the original driver). > > Pkg-gen/netmap is configured to receive only the frames directed to the > > physical MAC address of the interface. > > I don't know if and how a promiscuous mode can be enabled with netmap. > > > > Thanks, > > -Walter > > 2013/3/22 Tahir Rauf <tahir.rauf1@gmail.com> > > > >> Dear Walter, > >> > >> Can you please disable Rx/Tx pause frames on your NIC. You can use > >> "ethtool" to do so. Please run the example again after disabling Rx/Tx > >> frames and let me know about the results. > >> > >> Regards > >> > >> > >> On Thu, Mar 21, 2013 at 8:20 PM, Walter de Donato < > >> walter.dedonato@gmail.com> wrote: > >> > >>> Not that lucky since my first goal is to transmit packets... > >>> I also noticed that the same car d is configured with 4 rings on 32bit > >>> kernels and 16 rings on 64bit kernels. > >>> Do you have any idea to explain this difference? > >>> > >>> -Walter > >>> > >>> 2013/3/21 Tahir Rauf <tahir.rauf1@gmail.com> > >>> > >>>> Dear Walter, > >>>> > >>>> You are lucky to have* X540-AT2 :). *Mine is different (82598EB) and > >>>> can't receive packets properly. > >>>> Yes, my tx is working and is able to send packets on line rate. > >>>> > >>>> Regards > >>>> > >>>> > >>>> On Thu, Mar 21, 2013 at 8:04 PM, Walter de Donato < > >>>> walter.dedonato@gmail.com> wrote: > >>>> > >>>>> Here is the lshw output: > >>>>> > >>>>> *-network:0 > >>>>> description: Ethernet interface > >>>>> product: Ethernet Controller 10 Gigabit X540-AT2 > >>>>> vendor: Intel Corporation > >>>>> physical id: 0 > >>>>> bus info: pci@0000:0c:00.0 > >>>>> logical name: eth2 > >>>>> version: 01 > >>>>> serial: a0:36:9f:0d:f0:98 > >>>>> capacity: 1Gbit/s > >>>>> width: 64 bits > >>>>> clock: 33MHz > >>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list > rom > >>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation > >>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe > >>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef latency=0 > link=yes > >>>>> multicast=yes port=twisted pair > >>>>> resources: irq:17 memory:d8400000-d85fffff > >>>>> memory:d83fc000-d83fffff memory:fc500000-fc57ffff > memory:fc400000-fc4fffff > >>>>> *-network:1 > >>>>> description: Ethernet interface > >>>>> product: Ethernet Controller 10 Gigabit X540-AT2 > >>>>> vendor: Intel Corporation > >>>>> physical id: 0.1 > >>>>> bus info: pci@0000:0c:00.1 > >>>>> logical name: eth3 > >>>>> version: 01 > >>>>> serial: a0:36:9f:0d:f0:9a > >>>>> capacity: 1Gbit/s > >>>>> width: 64 bits > >>>>> clock: 33MHz > >>>>> capabilities: pm msi msix pciexpress vpd bus_master cap_list > rom > >>>>> ethernet physical tp 100bt-fd 1000bt-fd autonegotiation > >>>>> configuration: autonegotiation=on broadcast=yes driver=ixgbe > >>>>> driverversion=3.6.7-k duplex=full firmware=0x800002ef ip=192.168.78.2 > >>>>> latency=0 link=yes multicast=yes port=twisted pair > >>>>> resources: irq:16 memory:d8000000-d81fffff > >>>>> memory:d83f8000-d83fbfff memory:d8200000-d827ffff > >>>>> > >>>>> Did you check if in tx mode your pkt-gen effectively sends the > packets > >>>>> on the wire? > >>>>> What rates are you observing with it? > >>>>> > >>>>> -Walter > >>>>> > >>>>> 2013/3/21 Tahir Rauf <tahir.rauf1@gmail.com> > >>>>> > >>>>>> Dear walter, > >>>>>> > >>>>>> Can you please run following command to get your NIC information. > >>>>>> sudo lshw -class network > >>>>>> > >>>>>> I have *82598EB *10GB AF dual port network connection NIC and my rx > >>>>>> is not working. > >>>>>> > >>>>>> Regards > >>>>>> > >>>>>> > >>>>>> On Thu, Mar 21, 2013 at 7:46 PM, Walter de Donato < > >>>>>> walter.dedonato@gmail.com> wrote: > >>>>>> > >>>>>>> I tried the rx command and it seems to work: > >>>>>>> $ sudo ./pkt-gen -i eth3 -f rx -n 100000 -l 60 -w 1 > >>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1 > >>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 > >>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1 > >>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 > >>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:9a > >>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:9a starts at > a0:36:9f:d:f0:9a > >>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff > >>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at > ff:ff:ff:ff:ff:ff > >>>>>>> main [1401] map size is 334980 Kb > >>>>>>> main [1423] mapping 334980 Kbytes > >>>>>>> Receiving from eth3: 4 queues, 1 threads and 1 cpus. > >>>>>>> main [1477] Wait 1 secs for phy reset > >>>>>>> main [1479] Ready... > >>>>>>> main_thread [1078] 0 pps (0 pkts in 1001020 usec) > >>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0 > >>>>>>> main_thread [1078] 0 pps (0 pkts in 1001059 usec) > >>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0 > >>>>>>> main_thread [1078] 0 pps (0 pkts in 1001047 usec) > >>>>>>> receiver_body [833] waiting for initial packets, poll returns 0 0 > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001118 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001040 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001038 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001030 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001028 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001051 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001068 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001040 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001035 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001083 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001050 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001073 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001052 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001041 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001051 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001058 usec) > >>>>>>> main_thread [1078] 1 pps (1 pkts in 1001020 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001021 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001022 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001072 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001020 usec) > >>>>>>> main_thread [1078] 3 pps (3 pkts in 1001041 usec) > >>>>>>> main_thread [1078] 3 pps (3 pkts in 1001023 usec) > >>>>>>> main_thread [1078] 4 pps (4 pkts in 1001025 usec) > >>>>>>> main_thread [1078] 3 pps (3 pkts in 1001032 usec) > >>>>>>> main_thread [1078] 3 pps (3 pkts in 1001058 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001042 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001018 usec) > >>>>>>> main_thread [1078] 2 pps (2 pkts in 1001024 usec) > >>>>>>> main_thread [1078] 0 pps (0 pkts in 1001058 usec) > >>>>>>> Received 58 packets, in 29.16 seconds. > >>>>>>> Speed: 1.99 pps > >>>>>>> > >>>>>>> > >>>>>>> The proble is that on the other side I had to use ping instead of > >>>>>>> pkt-gen. > >>>>>>> > >>>>>>> If I try to use pkt-gen I don't receive any packet. > >>>>>>> I just checked that pkt-gen is not effectively sending the packets > on > >>>>>>> the wire, > >>>>>>> on the other side using tcpdump I don't receive any packet. > >>>>>>> > >>>>>>> So the problem on the tx side has not been completely solved in my > >>>>>>> case. > >>>>>>> Can you do the same test please? > >>>>>>> > >>>>>>> -Walter > >>>>>>> > >>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1@gmail.com> > >>>>>>> > >>>>>>>> Dear Walter, > >>>>>>>> > >>>>>>>> What about the Rx, Can you please try and inform me. > >>>>>>>> sudo ./pkt-gen -i eth2 -f rx -n 100000 -l 60 -w 1 > >>>>>>>> > >>>>>>>> Regards > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Mar 21, 2013 at 7:06 PM, Walter de Donato < > >>>>>>>> walter.dedonato@gmail.com> wrote: > >>>>>>>> > >>>>>>>>> Thanks Thair, > >>>>>>>>> > >>>>>>>>> with the fixed version the interface correctly attaches to the > >>>>>>>>> netmap driver and it is able to transmit packets. > >>>>>>>>> > >>>>>>>>> Anyway, I'm observing really poor performance: > >>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -n 100000 -l 60 -w 1 > >>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1 > >>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 > >>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1 > >>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 > >>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98 > >>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at > a0:36:9f:d:f0:98 > >>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff > >>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at > >>>>>>>>> ff:ff:ff:ff:ff:ff > >>>>>>>>> main [1401] map size is 334980 Kb > >>>>>>>>> main [1423] mapping 334980 Kbytes > >>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus. > >>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff) > >>>>>>>>> main [1477] Wait 1 secs for phy reset > >>>>>>>>> main [1479] Ready... > >>>>>>>>> sender_body [683] start > >>>>>>>>> main_thread [1078] 7101 pps (7108 pkts in 1001019 usec) > >>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001031 usec) > >>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001026 usec) > >>>>>>>>> main_thread [1078] 3483 pps (3487 pkts in 1001026 usec) > >>>>>>>>> main_thread [1078] 3340 pps (3343 pkts in 1001019 usec) > >>>>>>>>> main_thread [1078] 3455 pps (3459 pkts in 1001025 usec) > >>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001024 usec) > >>>>>>>>> main_thread [1078] 3494 pps (3498 pkts in 1001022 usec) > >>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001056 usec) > >>>>>>>>> main_thread [1078] 3488 pps (3492 pkts in 1001093 usec) > >>>>>>>>> main_thread [1078] 3474 pps (3478 pkts in 1001064 usec) > >>>>>>>>> main_thread [1078] 3485 pps (3489 pkts in 1001102 usec) > >>>>>>>>> main_thread [1078] 3479 pps (3483 pkts in 1001064 usec) > >>>>>>>>> main_thread [1078] 3335 pps (3339 pkts in 1001111 usec) > >>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001074 usec) > >>>>>>>>> main_thread [1078] 3484 pps (3488 pkts in 1001030 usec) > >>>>>>>>> main_thread [1078] 3470 pps (3474 pkts in 1001046 usec) > >>>>>>>>> main_thread [1078] 3486 pps (3490 pkts in 1001040 usec) > >>>>>>>>> main_thread [1078] 3482 pps (3486 pkts in 1001035 usec) > >>>>>>>>> main_thread [1078] 3278 pps (3281 pkts in 1001045 usec) > >>>>>>>>> main_thread [1078] 3476 pps (3480 pkts in 1001021 usec) > >>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001036 usec) > >>>>>>>>> main_thread [1078] 3477 pps (3481 pkts in 1001083 usec) > >>>>>>>>> main_thread [1078] 3493 pps (3497 pkts in 1001048 usec) > >>>>>>>>> main_thread [1078] 3353 pps (3357 pkts in 1001082 usec) > >>>>>>>>> main_thread [1078] 3490 pps (3494 pkts in 1001051 usec) > >>>>>>>>> main_thread [1078] 3465 pps (3469 pkts in 1001060 usec) > >>>>>>>>> main_thread [1078] 3008 pps (3011 pkts in 1001030 usec) > >>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001039 usec) > >>>>>>>>> Sent 100000 packets, 60 bytes each, in 28.58 seconds. > >>>>>>>>> Speed: 3.50 Kpps Bandwidth: 1.68 Mbps (raw 2.35 Mbps) > >>>>>>>>> > >>>>>>>>> With the rtl8169 1G driver (which they say to have the worst > >>>>>>>>> performance) > >>>>>>>>> I easily obtained Speed: 359.24Kpps. Bandwidth: 172.44Mbps with > 64 > >>>>>>>>> bytes packets. > >>>>>>>>> > >>>>>>>>> Also, if I increase the packet size this is what happens: > >>>>>>>>> $ sudo ./pkt-gen -i eth2 -f tx -t 1000000 -l 508 > >>>>>>>>> main [1270] -t deprecated, please use -f tx -n 1000000 > >>>>>>>>> extract_ip_range [136] extract IP range from 10.0.0.1 > >>>>>>>>> extract_ip_range [171] range is 10.0.0.1 0 to 10.0.0.1 0 > >>>>>>>>> extract_ip_range [136] extract IP range from 10.1.0.1 > >>>>>>>>> extract_ip_range [171] range is 10.1.0.1 0 to 10.1.0.1 0 > >>>>>>>>> extract_mac_range [177] extract MAC range from a0:36:9f:0d:f0:98 > >>>>>>>>> extract_mac_range [192] a0:36:9f:0d:f0:98 starts at > a0:36:9f:d:f0:98 > >>>>>>>>> extract_mac_range [177] extract MAC range from ff:ff:ff:ff:ff:ff > >>>>>>>>> extract_mac_range [192] ff:ff:ff:ff:ff:ff starts at > >>>>>>>>> ff:ff:ff:ff:ff:ff > >>>>>>>>> main [1401] map size is 334980 Kb > >>>>>>>>> main [1423] mapping 334980 Kbytes > >>>>>>>>> Sending on eth2: 4 queues, 1 threads and 1 cpus. > >>>>>>>>> 10.0.0.1 -> 10.1.0.1 (a0:36:9f:0d:f0:98 -> ff:ff:ff:ff:ff:ff) > >>>>>>>>> main [1477] Wait 2 secs for phy reset > >>>>>>>>> main [1479] Ready... > >>>>>>>>> sender_body [683] start > >>>>>>>>> main_thread [1078] 2042 pps (2044 pkts in 1001016 usec) > >>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001049 usec) > >>>>>>>>> sender_body [729] poll error/timeout on queue 0 > >>>>>>>>> main_thread [1078] 0 pps (0 pkts in 1001035 usec) > >>>>>>>>> main_thread [1096] ouch, thread 0 exited with error > >>>>>>>>> Sent 2044 packets, 508 bytes each, in -1363874673.38 seconds. > >>>>>>>>> Speed: -0.00 pps Bandwidth: -0.01 bps (raw -0.01 bps) > >>>>>>>>> > >>>>>>>>> Don't you experience the same issues? > >>>>>>>>> > >>>>>>>>> -Walter > >>>>>>>>> > >>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1@gmail.com> > >>>>>>>>> > >>>>>>>>>> Dear Walter, > >>>>>>>>>> > >>>>>>>>>> Please try following code. > >>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20130217-netmap.tgz > >>>>>>>>>> It will fix your problem with Ixgbe tx function. > >>>>>>>>>> > >>>>>>>>>> Please let me know, if you encounter any problem. > >>>>>>>>>> > >>>>>>>>>> Regards > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Mar 21, 2013 at 6:28 PM, Walter de Donato < > >>>>>>>>>> walter.dedonato@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> Dear Tahir, > >>>>>>>>>>> > >>>>>>>>>>> thank you for the prompt reply. > >>>>>>>>>>> I downloaded the source code from the official website: > >>>>>>>>>>> http://info.iet.unipi.it/~luigi/doc/20120813-netmap.tgz > >>>>>>>>>>> Can you please share with me the fix about the tx function? > >>>>>>>>>>> I was just starting to look at the patched code of the drivers. > >>>>>>>>>>> > >>>>>>>>>>> Thanks in advance, > >>>>>>>>>>> -Walter > >>>>>>>>>>> > >>>>>>>>>>> 2013/3/21 Tahir Rauf <tahir.rauf1@gmail.com> > >>>>>>>>>>> > >>>>>>>>>>>> Hi dedonato, > >>>>>>>>>>>> > >>>>>>>>>>>> I just observed that you are having trouble with tx function > of > >>>>>>>>>>>> pkt-gen. Whereas, we already have fixed that problem. We are > facing issues > >>>>>>>>>>>> with rx only now. > >>>>>>>>>>>> Please tell me that from where you get the code of netmap? > >>>>>>>>>>>> > >>>>>>>>>>>> Regards > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:49 PM, Tahir Rauf < > >>>>>>>>>>>> tahir.rauf1@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Dear Walter, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I tried to fix the problem but i don't have any success yet. > >>>>>>>>>>>>> Also if you are trying to fix this problem, please share with > >>>>>>>>>>>>> me that which path you are going to take for fixing. It > might be possible > >>>>>>>>>>>>> that I already have tried that path. In that case, I can > share my results > >>>>>>>>>>>>> with you, thus saving your time. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Mar 21, 2013 at 5:08 PM, <walter.dedonato@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Dear Tahir and Luigi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I have exactly the same problem with a similar hardware/os > >>>>>>>>>>>>>> configuration reported in the following. > >>>>>>>>>>>>>> Do you hav any uptade on this issue? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> -Walter > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Netmap version: 20120813 > >>>>>>>>>>>>>> OS: Ubuntu 12.04.2 LTS > >>>>>>>>>>>>>> Kernel: 3.2.0-38-generic-pae #61-Ubuntu SMP Tue Feb 19 > >>>>>>>>>>>>>> 12:39:51 UTC 2013 i686 i686 i386 GNU/Linux > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Network interfaces: > >>>>>>>>>>>>>> eth2 > >>>>>>>>>>>>>> 0c:00.0 Ethernet controller: Intel Corporation Ethernet > >>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01) > >>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t > >>>>>>>>>>>>>> Adapter > >>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 17 > >>>>>>>>>>>>>> Memory at d8400000 (64-bit, prefetchable) [size=2M] > >>>>>>>>>>>>>> Memory at d83fc000 (64-bit, prefetchable) [size=16K] > >>>>>>>>>>>>>> Expansion ROM at fc500000 [disabled] [size=512K] > >>>>>>>>>>>>>> Capabilities: [40] Power Management version 3 > >>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > >>>>>>>>>>>>>> 64bit+ > >>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > >>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00 > >>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data > >>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting > >>>>>>>>>>>>>> Capabilities: [140] Device Serial Number > >>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98 > >>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID > >>>>>>>>>>>>>> Interpretation (ARI) > >>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization > >>>>>>>>>>>>>> (SR-IOV) > >>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services > >>>>>>>>>>>>>> Kernel driver in use: ixgbe > >>>>>>>>>>>>>> Kernel modules: ixgbe > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> eth3 > >>>>>>>>>>>>>> 0c:00.1 Ethernet controller: Intel Corporation Ethernet > >>>>>>>>>>>>>> Controller 10 Gigabit X540-AT2 (rev 01) > >>>>>>>>>>>>>> Subsystem: Intel Corporation Ethernet 10G 2P X540-t > >>>>>>>>>>>>>> Adapter > >>>>>>>>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 16 > >>>>>>>>>>>>>> Memory at d8000000 (64-bit, prefetchable) [size=2M] > >>>>>>>>>>>>>> Memory at d83f8000 (64-bit, prefetchable) [size=16K] > >>>>>>>>>>>>>> Expansion ROM at d8200000 [disabled] [size=512K] > >>>>>>>>>>>>>> Capabilities: [40] Power Management version 3 > >>>>>>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > >>>>>>>>>>>>>> 64bit+ > >>>>>>>>>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > >>>>>>>>>>>>>> Capabilities: [a0] Express Endpoint, MSI 00 > >>>>>>>>>>>>>> Capabilities: [e0] Vital Product Data > >>>>>>>>>>>>>> Capabilities: [100] Advanced Error Reporting > >>>>>>>>>>>>>> Capabilities: [140] Device Serial Number > >>>>>>>>>>>>>> a0-36-9f-ff-ff-0d-f0-98 > >>>>>>>>>>>>>> Capabilities: [150] Alternative Routing-ID > >>>>>>>>>>>>>> Interpretation (ARI) > >>>>>>>>>>>>>> Capabilities: [160] Single Root I/O Virtualization > >>>>>>>>>>>>>> (SR-IOV) > >>>>>>>>>>>>>> Capabilities: [1d0] Access Control Services > >>>>>>>>>>>>>> Kernel driver in use: ixgbe > >>>>>>>>>>>>>> Kernel modules: ixgbe > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> After loading the modules this is the (interesting) lsmod > >>>>>>>>>>>>>> output: > >>>>>>>>>>>>>> $ lsmod > >>>>>>>>>>>>>> Module Size Used by > >>>>>>>>>>>>>> ixgbe 160150 0 > >>>>>>>>>>>>>> netmap_lin 102855 1 ixgbe > >>>>>>>>>>>>>> dca 14728 1 ixgbe > >>>>>>>>>>>>>> mdio 13559 1 ixgbe > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> And the following is the (interesting) dmesg output: > >>>>>>>>>>>>>> [175397.625200] 521.870604 netmap_new_obj_allocator [426] > >>>>>>>>>>>>>> objsize 1024 clustsize 4096 objects 4 > >>>>>>>>>>>>>> [175397.625501] 521.870912 netmap_new_obj_allocator [504] > >>>>>>>>>>>>>> Pre-allocated 128 clusters (4/512KB) for 'netmap_if' > >>>>>>>>>>>>>> [175397.625653] 521.871065 netmap_new_obj_allocator [426] > >>>>>>>>>>>>>> objsize 36864 clustsize 36864 objects 1 > >>>>>>>>>>>>>> [175397.629906] 521.875312 netmap_new_obj_allocator [504] > >>>>>>>>>>>>>> Pre-allocated 200 clusters (36/7200KB) for 'netmap_ring' > >>>>>>>>>>>>>> [175397.630064] 521.875477 netmap_new_obj_allocator [426] > >>>>>>>>>>>>>> objsize 2048 clustsize 4096 objects 2 > >>>>>>>>>>>>>> [175397.708572] 521.953976 netmap_new_obj_allocator [504] > >>>>>>>>>>>>>> Pre-allocated 50000 clusters (4/200000KB) for 'netmap_buf' > >>>>>>>>>>>>>> [175397.708737] 521.954147 netmap_memory_init [554] Have 512 > >>>>>>>>>>>>>> KB for interfaces, 7200 KB for rings and 195 MB for buffers > >>>>>>>>>>>>>> [175397.708885] netmap: loaded module with 202 Mbytes > >>>>>>>>>>>>>> [175418.013476] ixgbe: Intel(R) 10 Gigabit PCI Express > Network > >>>>>>>>>>>>>> Driver - version 3.6.7-k > >>>>>>>>>>>>>> [175418.013479] ixgbe: Copyright (c) 1999-2011 Intel > >>>>>>>>>>>>>> Corporation. > >>>>>>>>>>>>>> [175418.013531] ixgbe 0000:0c:00.0: PCI INT B -> GSI 17 > >>>>>>>>>>>>>> (level, low) -> IRQ 17 > >>>>>>>>>>>>>> [175418.013543] ixgbe 0000:0c:00.0: setting latency timer > to 64 > >>>>>>>>>>>>>> [175418.425782] ixgbe 0000:0c:00.0: irq 271 for MSI/MSI-X > >>>>>>>>>>>>>> [175418.425791] ixgbe 0000:0c:00.0: irq 272 for MSI/MSI-X > >>>>>>>>>>>>>> [175418.425798] ixgbe 0000:0c:00.0: irq 273 for MSI/MSI-X > >>>>>>>>>>>>>> [175418.425806] ixgbe 0000:0c:00.0: irq 274 for MSI/MSI-X > >>>>>>>>>>>>>> [175418.425813] ixgbe 0000:0c:00.0: irq 275 for MSI/MSI-X > >>>>>>>>>>>>>> [175418.425829] ixgbe 0000:0c:00.0: Multiqueue Enabled: Rx > >>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4 > >>>>>>>>>>>>>> [175418.485972] ixgbe 0000:0c:00.0: (PCI > Express:2.5GT/s:Width > >>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:98 > >>>>>>>>>>>>>> [175418.646372] ixgbe 0000:0c:00.0: MAC: 3, PHY: 3, PBA No: > >>>>>>>>>>>>>> G44743-006 > >>>>>>>>>>>>>> [175418.788262] ixgbe 0000:0c:00.0: Intel(R) 10 Gigabit > >>>>>>>>>>>>>> Network Connection > >>>>>>>>>>>>>> [175418.788285] ixgbe 0000:0c:00.1: PCI INT A -> GSI 16 > >>>>>>>>>>>>>> (level, low) -> IRQ 16 > >>>>>>>>>>>>>> [175418.788296] ixgbe 0000:0c:00.1: setting latency timer > to 64 > >>>>>>>>>>>>>> [175419.213734] ixgbe 0000:0c:00.1: irq 276 for MSI/MSI-X > >>>>>>>>>>>>>> [175419.213742] ixgbe 0000:0c:00.1: irq 277 for MSI/MSI-X > >>>>>>>>>>>>>> [175419.213750] ixgbe 0000:0c:00.1: irq 278 for MSI/MSI-X > >>>>>>>>>>>>>> [175419.213758] ixgbe 0000:0c:00.1: irq 279 for MSI/MSI-X > >>>>>>>>>>>>>> [175419.213766] ixgbe 0000:0c:00.1: irq 280 for MSI/MSI-X > >>>>>>>>>>>>>> [175419.213782] ixgbe 0000:0c:00.1: Multiqueue Enabled: Rx > >>>>>>>>>>>>>> Queue count = 4, Tx Queue count = 4 > >>>>>>>>>>>>>> [175419.273937] ixgbe 0000:0c:00.1: (PCI > Express:2.5GT/s:Width > >>>>>>>>>>>>>> x8) a0:36:9f:0d:f0:9a > >>>>>>>>>>>>>> [175419.434299] ixgbe 0000:0c:00.1: MAC: 3, PHY: 3, PBA No: > >>>>>>>>>>>>>> G44743-006 > >>>>>>>>>>>>>> [175419.576197] ixgbe 0000:0c:00.1: Intel(R) 10 Gigabit > >>>>>>>>>>>>>> Network Connection > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This is the pkt-gen output: > >>>>>>>>>>>>>> $ sudo ./pkt-gen -i eth2 tx -n 10000 > >>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.0.0.1 > >>>>>>>>>>>>>> extract_ip_range [129] 10.0.0.1 starts at 10.0.0.1 > >>>>>>>>>>>>>> extract_ip_range [119] extract IP range from 10.1.0.1 > >>>>>>>>>>>>>> extract_ip_range [129] 10.1.0.1 starts at 10.1.0.1 > >>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from > >>>>>>>>>>>>>> a0:36:9f:0d:f0:98 > >>>>>>>>>>>>>> extract_mac_range [150] a0:36:9f:0d:f0:98 starts at > >>>>>>>>>>>>>> a0:36:9f:d:f0:98 > >>>>>>>>>>>>>> extract_mac_range [135] extract MAC range from > >>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff > >>>>>>>>>>>>>> extract_mac_range [150] ff:ff:ff:ff:ff:ff starts at > >>>>>>>>>>>>>> ff:ff:ff:ff:ff:ff > >>>>>>>>>>>>>> main [1053] map size is 207712 Kb > >>>>>>>>>>>>>> main [1059] Unable to get if info for eth2 > >>>>>>>>>>>>>> main [1066] bad nthreads 1, have 0 queues > >>>>>>>>>>>>>> main [1075] mmapping 207712 Kbytes > >>>>>>>>>>>>>> main [1094] Unable to register interface eth2 > >>>>>>>>>>>>>> Receiving from eth2: 0 queues, 1 threads and 1 cpus. > >>>>>>>>>>>>>> main [1128] Wait 2 secs for phy reset > >>>>>>>>>>>>>> main [1130] Ready... > >>>>>>>>>>>>>> main [1181] Unable to register eth2 > >>>>>>>>>>>>>> main [1242] 0 pps > >>>>>>>>>>>>>> Received 0 packets, in 0.00 seconds. > >>>>>>>>>>>>>> Speed: -nanpps. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dmesg after trying to use pkt-gen on eth2: > >>>>>>>>>>>>>> [175537.838503] ADDRCONF(NETDEV_UP): eth2: link is not ready > >>>>>>>>>>>>>> [175542.802288] ixgbe 0000:0c:00.0: eth2: NIC Link is Up 10 > >>>>>>>>>>>>>> Gbps, Flow Control: RX/TX > >>>>>>>>>>>>>> [175542.802607] ADDRCONF(NETDEV_CHANGE): eth2: link becomes > >>>>>>>>>>>>>> ready > >>>>>>>>>>>>>> [175553.256029] eth2: no IPv6 routers present > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Tahir Rauf > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> Tahir Rauf > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Tahir Rauf > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Tahir Rauf > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Tahir Rauf > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Tahir Rauf > >>>> > >>>> > >>> > >> > >> > >> -- > >> Tahir Rauf > >> > >> > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPRxaNSZBe0zPybwwyS4AtOhQ1Z1U5Ly4ZPDhAtFm7s%2BP84zgw>