Date: Sat, 23 Jan 2016 19:10:33 -0800 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Marcus Cenzatti <cenzatti@hush.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Navdeep Parhar <nparhar@gmail.com> Subject: solved: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX Message-ID: <CA%2BhQ2%2Bg7_haaXLFjMuG00ANsUkFdyGzFQyjT4NYVBmPY-vECBg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Thanks for re-running the experiments.
I am changing the subject so that in the archives it is clear
that the chelsio card works fine.
Overall the tests confirm that whenever you hit the host stack you are boun=
d
to the poor performance of the latter. The problem does not appear using in=
tel
as a receiver because on the intel card netmap mode disables the host stack=
.
More comments on the experiments:
The only meaningful test is the one where you use the DMAC of the ncxl0 por=
t:
SENDER: ./pkt-gen -i ix0 -f tx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c=
1
in the other experiment you transmit broadcast frames and hit the network s=
tack.
ARP etc do not matter since tx and rx are directly connected.
On the receiver you do not need to specify addresses:
RECEIVER: ./pkt-gen -i ncxl0 -f rx
The numbers in netstat are clearly rounded, so 15M is probably 14.88M
(line rate), and 3.7M that you see correctly represents the difference
between incoming and received packets.
The fact that you see drops may be related to the NIC being unable to
replenish the queue fast enough, which in turn may be a hardware or a
software (netmap) issue.
You may try experiment with shorter batches on the receive side
(say, -b 64 or less) and see if you have better results.
A short batch replenishes the rx queue more frequently, but it is
not a conclusive experiment because there is an optimization in
the netmap poll code which, as an unintended side effect, replenishes
the queue less often than it should.
For a conclusive experiment you should grab the netmap code from
github.com/luigirizzo/netmap and use pkt-gen-b which
uses busy wait and works around the poll "optimization"
thanks again for investigating the issue.
cheers
luigi
On Sat, Jan 23, 2016 at 6:40 PM, Marcus Cenzatti <cenzatti@hush.com> wrote:
>
>
>
> dear gentlemen,
>
> i started over in the end of the day, rebooted everything and removed all=
tweeks (had no sysctl tweeks on the previous output but had removed tx|rx =
csum and tso from both intel and chelsio).
>
> mr Adrian Chadd suggested to shut cxl0 down but if I ifconfig cxl0 down b=
oth cxl0 and ncxl0 will be down, anyway I removed IP address from cxl0 and =
let only ncxl0 addressed.
>
> I have new numbers:
>
> - MAC address based tests: 14Mpps TX =3D> 11Mpps RX (much better, finally=
, but still lower than Intel-Intel, not a problem at all)
> - IP address based tests: 14Mpps TX =3D> 300Kpps RX (kernel rate possibly=
, but on netmap mode, why? a problem for sure)
>
> on both experiments we actually had no error pointed on netstat or sysctl=
s, and actual RX speed observed on netstat on the second test was 15Mpps.
>
> here is the full transcript for the two session tests with stats outputs.
>
> things i could not understand are:
>
> 1) on mac address tests the sender (tx) pkt-gen shows:
>
> Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
> 10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1)
>
> where those ip addresses come from? same is observed on rx:
>
> 529.477028 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
> 529.477042 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
>
> 2) on ip address tests, the sender (tx/intel) pkt-gen shows:
>
> 172.16.0.2 -> 172.16.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
>
> does this mean we are transmitting to broadcast?
>
> the receiver (chelsio/tx) shows this, that makes much more sense:
>
> 500.757057 main [1715] interface is ncxl0
> 500.757321 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
> 500.757348 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
> 501.007425 main [1910] mapped 334980KB at 0x801c00000
>
> so I could not explain why I had 800Kpps rates on chelsio when I had that=
tweeks and after rebooting i have a much more decent 11Mpps rate but i cou=
ld confirm, if I remove the tweeks I get 800K-1.2M pps again on MAC address=
tests.
>
> since cxl0 is not addressed and ncxl0 will only work in netmap mode i bel=
ieve it will never arp reply, therefore the netmap application should arp r=
eply when someone arp requests the actual mac address for a given ip addres=
s, will pkt-gen arp reply? because this could, maybe, explain the destinati=
on part for 00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff (but source mac address =
should be correct)
>
> just to remember, when I test intel-intel I get the same 14Mpps RX and TX=
rates on both IP address test as well as my previously wrong (using -s|-d =
instead of -S|-D pkt-gen options) mac address tests, but i don't remember i=
f mac address were shown like that
>
> luigi, the sysctl you asked navdeep about, sysctl dev.ncxl.0.stats, does =
not exist, only sysctl dev.cxl.0.stats exists and I could not find other st=
ats similar to dev.t5nex.0.misc.tp_err_stats so probably it's not different=
betweek cxl and ncxl
>
> =3D=3D=3D=3D FULL TESTS TRANSCRIPT =3D=3D=3D=3D
>
> intel# ifconfig -v ix0
> ix0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 15=
00
> options=3D8406b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TS=
O6,LRO,VLAN_HWTSO>
> ether 00:07:e9:44:d2:ba
> inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-SR (10Gbase-SR <full-duplex,rxpause,txpau=
se>)
> status: active
> plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
> vendor: FINISAR CORP. PN: FTLX8571D3BCL-FC SN: AM31TRU DATE: 2012=
-01-21
> module temperature: 34.98 C Voltage: 3.28 Volts
> RX: 0.50 mW (-2.98 dBm) TX: 0.58 mW (-2.33 dBm)
> intel# ./pkt-gen -i ix0 -f tx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:c1
> 268.505741 main [1715] interface is ix0
> 268.505795 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
> 268.505806 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
> 268.611395 main [1910] mapped 334980KB at 0x801c00000
> Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
> 10.0.0.1 -> 10.1.0.1 (00:07:e9:44:d2:ba -> 00:07:43:33:8d:c1)
> 268.611467 main [1994] Sending 512 packets every 0.000000000 s
> 268.611473 main [1996] Wait 2 secs for phy reset
> 270.617971 main [1998] Ready...
> 270.618120 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
> 270.635480 sender_body [1073] start, fd 4 main_fd 3
> 270.670823 sender_body [1142] drop copy
> 271.637012 main_thread [1512] 14461469 pps (14484000 pkts in 1001558 usec=
)
> 272.647011 main_thread [1512] 14880527 pps (15029317 pkts in 1009999 usec=
)
> 273.648511 main_thread [1512] 14884638 pps (14906965 pkts in 1001500 usec=
)
> 274.649506 main_thread [1512] 14883795 pps (14898619 pkts in 1000996 usec=
)
> 275.650509 main_thread [1512] 14879377 pps (14894301 pkts in 1001003 usec=
)
> 276.651511 main_thread [1512] 14883276 pps (14898174 pkts in 1001001 usec=
)
> 277.652512 main_thread [1512] 14880941 pps (14895852 pkts in 1001002 usec=
)
> 278.653517 main_thread [1512] 14879125 pps (14894079 pkts in 1001005 usec=
)
> 279.654525 main_thread [1512] 14887529 pps (14902431 pkts in 1001001 usec=
)
> 280.656518 main_thread [1512] 14864554 pps (14894268 pkts in 1001999 usec=
)
> 281.657012 main_thread [1512] 14882656 pps (14890008 pkts in 1000494 usec=
)
> 282.658506 main_thread [1512] 14881346 pps (14903594 pkts in 1001495 usec=
)
> 283.664016 main_thread [1512] 14882829 pps (14964819 pkts in 1005509 usec=
)
> 284.690513 main_thread [1512] 14880361 pps (15274661 pkts in 1026498 usec=
)
> 285.691504 main_thread [1512] 14877683 pps (14892412 pkts in 1000990 usec=
)
> 286.692143 main_thread [1512] 14885234 pps (14894761 pkts in 1000640 usec=
)
> 287.693510 main_thread [1512] 14885267 pps (14905615 pkts in 1001367 usec=
)
> 288.694010 main_thread [1512] 14879692 pps (14887132 pkts in 1000500 usec=
)
> ^C289.189696 sigint_h [328] received control-C on thread 0x801806800
> 289.189750 sender_body [1171] flush tail 19 head 19 on thread 0x801806800
> 289.189790 sender_body [1179] pending tx tail 72 head 1105 on ring 0
> 289.189823 sender_body [1179] pending tx tail 154 head 1105 on ring 0
> 289.189848 sender_body [1179] pending tx tail 222 head 1105 on ring 0
> 289.189871 sender_body [1179] pending tx tail 274 head 1105 on ring 0
> 289.189901 sender_body [1179] pending tx tail 338 head 1105 on ring 0
> 289.700237 main_thread [1512] 7332017 pps (7377666 pkts in 1006226 usec)
> Sent 275688674 packets, 60 bytes each, in 18.55 seconds.
> Speed: 14.86 Mpps Bandwidth: 7.13 Gbps (raw 9.98 Gbps)
> intel#
>
>
>
> chelsio# ./pkt-gen -i ncxl0 -f rx -S 00:07:e9:44:d2:ba -D 00:07:43:33:8d:=
c1
> 529.476963 main [1715] interface is ncxl0
> 529.477028 extract_ip_range [291] range is 10.0.0.1:0 to 10.0.0.1:0
> 529.477042 extract_ip_range [291] range is 10.1.0.1:0 to 10.1.0.1:0
> 529.535750 main [1910] mapped 334980KB at 0x801c00000
> Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
> 529.535829 main [1996] Wait 2 secs for phy reset
> 531.537267 main [1998] Ready...
> 531.537399 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
> 531.555050 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
> 532.555628 main_thread [1512] 11173691 pps (11180373 pkts in 1000598 usec=
)
> 533.556129 main_thread [1512] 11175384 pps (11180983 pkts in 1000501 usec=
)
> 534.567273 main_thread [1512] 11176345 pps (11300905 pkts in 1011145 usec=
)
> 535.568126 main_thread [1512] 11175813 pps (11185335 pkts in 1000852 usec=
)
> 536.569623 main_thread [1512] 11174668 pps (11191408 pkts in 1001498 usec=
)
> 537.580127 main_thread [1512] 11176569 pps (11293968 pkts in 1010504 usec=
)
> (..)
>
> 739.193065 main_thread [1512] 11173118 pps (11177744 pkts in 1000414 usec=
)
> 740.194062 main_thread [1512] 11175690 pps (11186832 pkts in 1000997 usec=
)
> 741.202063 main_thread [1512] 11176354 pps (11265776 pkts in 1008001 usec=
)
> 742.203564 main_thread [1512] 11176509 pps (11193296 pkts in 1001502 usec=
)
> 743.204559 main_thread [1512] 11175825 pps (11186945 pkts in 1000995 usec=
)
> ^C743.392524 sigint_h [328] received control-C on thread 0x801806800
> 744.212517 main_thread [1512] 2080802 pps (2097359 pkts in 1007957 usec)
> Received 2367310284 packets, in 211.84 seconds.
> Speed: 11.18 Mpps
>
>
> chelsio# netstat -hd -I ncxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0=
0 0 0
> ncxl0 - 172.16.0.0 172.16.0.1 158M - - 0=
- - -
> chelsio# netstat -hd -I cxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 7.1G 0 669M 0=
0 0 0
> chelsio# netstat -hdwi 1 -l ncxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 3.7M 908M 4 0 660 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 9 0 1.2K 0 0
> 15M 0 3.7M 908M 6 0 1.1K 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> ^C
>
> chelsio# netstat -hdwi 1 -l cxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 3.7M 908M 4 0 660 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 6 0 763 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 4 0 612 0 0
> 15M 0 3.7M 908M 9 0 1.0K 0 0
> ^C
>
> chelsio# netstat -hd -I ncxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0=
0 0 0
>
> ncxl0 - 172.16.0.0 172.16.0.1 39M - - 0=
- - -
> chelsio#
> chelsio# netstat -hd -I cxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 6.0G 1 612k 0=
0 0 0
> chelsio# netstat -hdwi 1 -l ncxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 0 908M 8 0 1.1K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 15 0 1.8K 0 0
> 15M 0 0 908M 10 0 1.2K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 6 0 822 0 0
> 15M 0 0 908M 6 0 826 0 0
> ^C
> chelsio# netstat -hdwi 1 -l cxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 0 908M 8 0 1.0K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 5 0 756 0 0
> 15M 0 0 908M 10 0 1.2K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 4 0 604 0 0
> ^C
>
>
>
> chelsio# sysctl -n dev.t5nex.0.misc.tp_err_stats
> channel 0 channel 1 channel 2 channel 3
> macInErrs: 0 0 0 0
> hdrInErrs: 0 0 0 0
> tcpInErrs: 0 0 0 0
> tcp6InErrs: 0 0 0 0
> tnlCongDrops: 42862617 0 0 0
> tnlTxDrops: 0 0 0 0
> ofldVlanDrops: 0 0 0 0
> ofldChanDrops: 0 0 0 0
>
> ofldNoNeigh: 0
> ofldCongDefer: 0
>
> chelsio# sysctl dev.cxl.0.stats
> dev.cxl.0.stats.rx_trunc3: 0
> dev.cxl.0.stats.rx_trunc2: 0
> dev.cxl.0.stats.rx_trunc1: 0
> dev.cxl.0.stats.rx_trunc0: 0
> dev.cxl.0.stats.rx_ovflow3: 0
> dev.cxl.0.stats.rx_ovflow2: 0
> dev.cxl.0.stats.rx_ovflow1: 0
> dev.cxl.0.stats.rx_ovflow0: 853083783
> dev.cxl.0.stats.rx_ppp7: 0
> dev.cxl.0.stats.rx_ppp6: 0
> dev.cxl.0.stats.rx_ppp5: 0
> dev.cxl.0.stats.rx_ppp4: 0
> dev.cxl.0.stats.rx_ppp3: 0
> dev.cxl.0.stats.rx_ppp2: 0
> dev.cxl.0.stats.rx_ppp1: 0
> dev.cxl.0.stats.rx_ppp0: 0
> dev.cxl.0.stats.rx_pause: 0
> dev.cxl.0.stats.rx_frames_1519_max: 0
> dev.cxl.0.stats.rx_frames_1024_1518: 0
> dev.cxl.0.stats.rx_frames_512_1023: 0
> dev.cxl.0.stats.rx_frames_256_511: 0
> dev.cxl.0.stats.rx_frames_128_255: 0
> dev.cxl.0.stats.rx_frames_65_127: 0
> dev.cxl.0.stats.rx_frames_64: 9778077198
> dev.cxl.0.stats.rx_runt: 0
> dev.cxl.0.stats.rx_symbol_err: 0
> dev.cxl.0.stats.rx_len_err: 0
> dev.cxl.0.stats.rx_fcs_err: 0
> dev.cxl.0.stats.rx_jabber: 0
> dev.cxl.0.stats.rx_too_long: 0
> dev.cxl.0.stats.rx_ucast_frames: 5365842946
> dev.cxl.0.stats.rx_mcast_frames: 0
> dev.cxl.0.stats.rx_bcast_frames: 4412237958
> dev.cxl.0.stats.rx_frames: 9778082382
> dev.cxl.0.stats.rx_octets: 625797304704
> dev.cxl.0.stats.tx_ppp7: 0
> dev.cxl.0.stats.tx_ppp6: 0
> dev.cxl.0.stats.tx_ppp5: 0
> dev.cxl.0.stats.tx_ppp4: 0
> dev.cxl.0.stats.tx_ppp3: 0
> dev.cxl.0.stats.tx_ppp2: 0
> dev.cxl.0.stats.tx_ppp1: 0
> dev.cxl.0.stats.tx_ppp0: 0
> dev.cxl.0.stats.tx_pause: 0
> dev.cxl.0.stats.tx_drop: 0
> dev.cxl.0.stats.tx_frames_1519_max: 0
> dev.cxl.0.stats.tx_frames_1024_1518: 0
> dev.cxl.0.stats.tx_frames_512_1023: 0
> dev.cxl.0.stats.tx_frames_256_511: 0
> dev.cxl.0.stats.tx_frames_128_255: 0
> dev.cxl.0.stats.tx_frames_65_127: 0
> dev.cxl.0.stats.tx_frames_64: 0
> dev.cxl.0.stats.tx_error_frames: 0
> dev.cxl.0.stats.tx_ucast_frames: 0
> dev.cxl.0.stats.tx_mcast_frames: 0
> dev.cxl.0.stats.tx_bcast_frames: 0
> dev.cxl.0.stats.tx_frames: 0
> dev.cxl.0.stats.tx_octets: 0
> dev.cxl.0.stats.tx_parse_error: 0
> chelsio# sysctl dev.ncxl.0.stats
>
> =3D=3D=3D> IP based tests:
>
> intel# ./pkt-gen -i ix0 -f tx -s 172.16.0.2 -d 172.16.0.1
> 150.673411 main [1715] interface is ix0
> 150.673545 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
> 150.673560 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
> 150.779091 main [1910] mapped 334980KB at 0x801c00000
> Sending on netmap:ix0: 8 queues, 1 threads and 1 cpus.
> 172.16.0.2 -> 172.16.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
> 150.779164 main [1994] Sending 512 packets every 0.000000000 s
> 150.779169 main [1996] Wait 2 secs for phy reset
> 152.846385 main [1998] Ready...
> 152.846515 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1
> 152.863946 sender_body [1073] start, fd 4 main_fd 3
> 152.898386 sender_body [1142] drop copy
> 153.864749 main_thread [1512] 14477663 pps (14489636 pkts in 1000827 usec=
)
> 154.868750 main_thread [1512] 14879784 pps (14939318 pkts in 1004001 usec=
)
> 155.873246 main_thread [1512] 14883091 pps (14950005 pkts in 1004496 usec=
)
> 156.874753 main_thread [1512] 14883859 pps (14906289 pkts in 1001507 usec=
)
> (...)
>
> chelsio# ./pkt-gen -i ncxl0 -f rx -s 172.16.0.2 -d 172.16.0.1
> 500.757057 main [1715] interface is ncxl0
> 500.757321 extract_ip_range [291] range is 172.16.0.2:0 to 172.16.0.2:0
> 500.757348 extract_ip_range [291] range is 172.16.0.1:0 to 172.16.0.1:0
> 501.007425 main [1910] mapped 334980KB at 0x801c00000
> Receiving from netmap:ncxl0: 2 queues, 1 threads and 1 cpus.
> 501.007511 main [1996] Wait 2 secs for phy reset
> 503.015342 main [1998] Ready...
> 503.015487 nm_open [456] overriding ifname ncxl0 ringid 0x0 flags 0x1
> 503.033543 receiver_body [1247] reading from netmap:ncxl0 fd 4 main_fd 3
> 504.034782 main_thread [1512] 303341 pps (303727 pkts in 1001273 usec)
> 505.035783 main_thread [1512] 301714 pps (302016 pkts in 1001000 usec)
> 506.036779 main_thread [1512] 302355 pps (302656 pkts in 1000996 usec)
> 507.037780 main_thread [1512] 301651 pps (301953 pkts in 1001002 usec)
> 508.038783 main_thread [1512] 301922 pps (302225 pkts in 1001002 usec)
> 509.039780 main_thread [1512] 301633 pps (301934 pkts in 1000997 usec)
> 510.040776 main_thread [1512] 304081 pps (304384 pkts in 1000996 usec)
> 511.043280 main_thread [1512] 305602 pps (306367 pkts in 1002504 usec)
> 512.044780 main_thread [1512] 305207 pps (305665 pkts in 1001500 usec)
> 513.045780 main_thread [1512] 305103 pps (305408 pkts in 1001001 usec)
> 514.056282 main_thread [1512] 304894 pps (308096 pkts in 1010502 usec)
> 515.057781 main_thread [1512] 305015 pps (305472 pkts in 1001499 usec)
> 516.058780 main_thread [1512] 305935 pps (306240 pkts in 1000998 usec)
> 517.066781 main_thread [1512] 305333 pps (307776 pkts in 1008001 usec)
> 518.073779 main_thread [1512] 304556 pps (306688 pkts in 1006999 usec)
> 519.081781 main_thread [1512] 304190 pps (306624 pkts in 1008002 usec)
> 520.087780 main_thread [1512] 304351 pps (306177 pkts in 1005998 usec)
> 521.088778 main_thread [1512] 305102 pps (305407 pkts in 1000999 usec)
> 522.089775 main_thread [1512] 305232 pps (305536 pkts in 1000997 usec)
> 523.090281 main_thread [1512] 305606 pps (305760 pkts in 1000505 usec)
> 524.091280 main_thread [1512] 305391 pps (305696 pkts in 1000999 usec)
> 525.093787 main_thread [1512] 305986 pps (306753 pkts in 1002507 usec)
> 526.101276 main_thread [1512] 305424 pps (307711 pkts in 1007489 usec)
> 527.102772 main_thread [1512] 305272 pps (305729 pkts in 1001496 usec)
> 528.106277 main_thread [1512] 305122 pps (306192 pkts in 1003506 usec)
> 529.107778 main_thread [1512] 305253 pps (305711 pkts in 1001501 usec)
> 523.090281 main_thread [1512] 305606 pps (305760 pkts in 1000505 usec)
> 524.091280 main_thread [1512] 305391 pps (305696 pkts in 1000999 usec)
> 525.093787 main_thread [1512] 305986 pps (306753 pkts in 1002507 usec)
> 526.101276 main_thread [1512] 305424 pps (307711 pkts in 1007489 usec)
> 527.102772 main_thread [1512] 305272 pps (305729 pkts in 1001496 usec)
> 528.106277 main_thread [1512] 305122 pps (306192 pkts in 1003506 usec)
> 529.107778 main_thread [1512] 305253 pps (305711 pkts in 1001501 usec)
> 530.110774 main_thread [1512] 304432 pps (305344 pkts in 1002995 usec)
> 531.111774 main_thread [1512] 304847 pps (305152 pkts in 1001000 usec)
> 532.115782 main_thread [1512] 305209 pps (306432 pkts in 1004008 usec)
> 533.122273 main_thread [1512] 305092 pps (307072 pkts in 1006491 usec)
> 534.129776 main_thread [1512] 304277 pps (306560 pkts in 1007504 usec)
> 535.130272 main_thread [1512] 305193 pps (305344 pkts in 1000496 usec)
> 536.135271 main_thread [1512] 304414 pps (305936 pkts in 1004999 usec)
> 537.136272 main_thread [1512] 305279 pps (305584 pkts in 1001000 usec)
> 538.137772 main_thread [1512] 305334 pps (305792 pkts in 1001500 usec)
> 539.138606 main_thread [1512] 304897 pps (305152 pkts in 1000835 usec)
> 540.140772 main_thread [1512] 305195 pps (305856 pkts in 1002165 usec)
> 541.148272 main_thread [1512] 304786 pps (307072 pkts in 1007501 usec)
> 542.149770 main_thread [1512] 304773 pps (305230 pkts in 1001498 usec)
> 543.153270 main_thread [1512] 305859 pps (306930 pkts in 1003500 usec)
> 544.154357 main_thread [1512] 305014 pps (305345 pkts in 1001086 usec)
> 545.156774 main_thread [1512] 304990 pps (305727 pkts in 1002418 usec)
> ^C546.157076 sigint_h [328] received control-C on thread 0x801806800
> 546.157273 main_thread [1512] 305512 pps (305664 pkts in 1000499 usec)
> Received 13138095 packets, in 43.12 seconds.
> Speed: 304.66 Kpps
>
> chelsio# netstat -hd -I ncxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> ncxl0 1.5K <Link#4> 00:07:43:33:8d:c1 0 0 0 0=
0 0 0
>
> ncxl0 - 172.16.0.0 172.16.0.1 39M - - 0=
- - -
> chelsio#
> chelsio# netstat -hd -I cxl0
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts=
Oerrs Coll Drop
> cxl0 1.5K <Link#3> 00:07:43:33:8d:c0 6.0G 1 612k 0=
0 0 0
> chelsio# netstat -hdwi 1 -l ncxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 0 908M 8 0 1.1K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 15 0 1.8K 0 0
> 15M 0 0 908M 10 0 1.2K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 6 0 822 0 0
> 15M 0 0 908M 6 0 826 0 0
> ^C
> chelsio# netstat -hdwi 1 -l cxl0
> input (Total) output
> packets errs idrops bytes packets errs bytes colls drop=
s
> 15M 0 0 908M 8 0 1.0K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 5 0 756 0 0
> 15M 0 0 908M 10 0 1.2K 0 0
> 15M 0 0 908M 4 0 604 0 0
> 15M 0 0 908M 4 0 604 0 0
> ^C
>
> chelsio# sysctl -n dev.t5nex.0.misc.tp_err_stats
> channel 0 channel 1 channel 2 channel 3
> macInErrs: 0 0 0 0
> hdrInErrs: 0 0 0 0
> tcpInErrs: 0 0 0 0
> tcp6InErrs: 0 0 0 0
> tnlCongDrops: 612276 0 0 0
> tnlTxDrops: 0 0 0 0
> ofldVlanDrops: 0 0 0 0
> ofldChanDrops: 0 0 0 0
>
> ofldNoNeigh: 0
> ofldCongDefer: 0
>
> chelsio# sysctl dev.ncxl.0.stats
> sysctl: unknown oid 'dev.ncxl.0.stats'
> Exit 1
> chelsio# sysctl dev.cxl.0.stats
> dev.cxl.0.stats.rx_trunc3: 0
> dev.cxl.0.stats.rx_trunc2: 0
> dev.cxl.0.stats.rx_trunc1: 0
> dev.cxl.0.stats.rx_trunc0: 0
> dev.cxl.0.stats.rx_ovflow3: 0
> dev.cxl.0.stats.rx_ovflow2: 0
> dev.cxl.0.stats.rx_ovflow1: 0
> dev.cxl.0.stats.rx_ovflow0: 0
> dev.cxl.0.stats.rx_ppp7: 0
> dev.cxl.0.stats.rx_ppp6: 0
> dev.cxl.0.stats.rx_ppp5: 0
> dev.cxl.0.stats.rx_ppp4: 0
> dev.cxl.0.stats.rx_ppp3: 0
> dev.cxl.0.stats.rx_ppp2: 0
> dev.cxl.0.stats.rx_ppp1: 0
> dev.cxl.0.stats.rx_ppp0: 0
> dev.cxl.0.stats.rx_pause: 0
> dev.cxl.0.stats.rx_frames_1519_max: 0
> dev.cxl.0.stats.rx_frames_1024_1518: 0
> dev.cxl.0.stats.rx_frames_512_1023: 0
> dev.cxl.0.stats.rx_frames_256_511: 0
> dev.cxl.0.stats.rx_frames_128_255: 0
> dev.cxl.0.stats.rx_frames_65_127: 0
> dev.cxl.0.stats.rx_frames_64: 8216274634
> dev.cxl.0.stats.rx_runt: 0
> dev.cxl.0.stats.rx_symbol_err: 0
> dev.cxl.0.stats.rx_len_err: 0
> dev.cxl.0.stats.rx_fcs_err: 1
> dev.cxl.0.stats.rx_jabber: 0
> dev.cxl.0.stats.rx_too_long: 0
> dev.cxl.0.stats.rx_ucast_frames: 0
> dev.cxl.0.stats.rx_mcast_frames: 0
> dev.cxl.0.stats.rx_bcast_frames: 8216279461
> dev.cxl.0.stats.rx_frames: 8216280197
> dev.cxl.0.stats.rx_octets: 525841964992
> dev.cxl.0.stats.tx_ppp7: 0
> dev.cxl.0.stats.tx_ppp6: 0
> dev.cxl.0.stats.tx_ppp5: 0
> dev.cxl.0.stats.tx_ppp4: 0
> dev.cxl.0.stats.tx_ppp3: 0
> dev.cxl.0.stats.tx_ppp2: 0
> dev.cxl.0.stats.tx_ppp1: 0
> dev.cxl.0.stats.tx_ppp0: 0
> dev.cxl.0.stats.tx_pause: 0
> dev.cxl.0.stats.tx_drop: 0
> dev.cxl.0.stats.tx_frames_1519_max: 0
> dev.cxl.0.stats.tx_frames_1024_1518: 0
> dev.cxl.0.stats.tx_frames_512_1023: 0
> dev.cxl.0.stats.tx_frames_256_511: 0
> dev.cxl.0.stats.tx_frames_128_255: 0
> dev.cxl.0.stats.tx_frames_65_127: 0
> dev.cxl.0.stats.tx_frames_64: 0
> dev.cxl.0.stats.tx_error_frames: 0
> dev.cxl.0.stats.tx_ucast_frames: 0
> dev.cxl.0.stats.tx_mcast_frames: 0
> dev.cxl.0.stats.tx_bcast_frames: 0
> dev.cxl.0.stats.tx_frames: 0
> dev.cxl.0.stats.tx_octets: 0
> dev.cxl.0.stats.tx_parse_error: 0
>
>
--=20
-----------------------------------------+-------------------------------
Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione
http://www.iet.unipi.it/~luigi/ . Universita` di Pisa
TEL +39-050-2217533 . via Diotisalvi 2
Mobile +39-338-6809875 . 56122 PISA (Italy)
-----------------------------------------+-------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BhQ2%2Bg7_haaXLFjMuG00ANsUkFdyGzFQyjT4NYVBmPY-vECBg>