Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 2 Dec 2016 09:55:03 +0100
From:      Jakub Palider <jpa@semihalf.com>
To:        Jordan Caraballo <jordancaraballo87@gmail.com>
Cc:        freebsd-net@freebsd.org, freebsd-hpc@freebsd.org,  freebsd-performance@freebsd.org
Subject:   Re: netmap, netmap-fwd, and how many M packets-per-second?
Message-ID:  <CAL7QUyNtfVwjo640yb2PW-UzY%2BL17vghwwjkdNLyMXsyz5SQ%2Bg@mail.gmail.com>
In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com>
References:  <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi,

I have run some performance tests but not as router, which is your case, so
some information might not be relevant:
0. validate your hw stats with what you get from the output
1. Try using FLOWTABLE option in your kernel config - this may kill your
machine in -CURRENT, but it's worth checking
2. MTU set to some reasonably small value (~120) shall prevent from
consolidating small packets into big frame
3. disable all kernel tracing (INVARIANTS, WITNESS, etc)
4. I read you tuned the system, but make sure there are enough resources
5. Check that context switches are not killing it

Regardless on the above you might want profile the system with dtrace and
export results to flamegraph, this will give you an idea what going on in
the system and possibly locate the cause of the problem. It's definitely
worth the effort.

Jakub

On Fri, Dec 2, 2016 at 12:55 AM, Jordan Caraballo <
jordancaraballo87@gmail.com> wrote:

> Feedback and/or tips and tricks more than welcome.
>
> We are trying to process huge amounts of small (64 bytes) pps through a
> router. So far results have not been as we expected. We have tested
> FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without
> netmap. Based on netmap documentation we were expecting about 5.0M pps;
> alongside with the routing improvements from the freebsd routing
> proposal a total of 12.0M.
>
> Server Description:
>
> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores =
per
> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot.
>
> BIOS tweaks:
>
> Hyperthreading (or Logical Processors) is turned off.
>
> Current results are shown below. Additional configurations can be given
> upon request.
>
> Test Environment:
> 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one
> equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2).
>
> Script that execute the following on each host.
>
> #!/usr/local/bin/bash
> # Iterate through ports and start tests
> for ((i=3D1;i<=3DPORTS;i++)); do
>     PORT=3D$(($PORT+1))
>     iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS
> -p$PORT &
>     #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT &
> done
>
> # FreeBSD 10.3 - 4 streams to 80 ports from each client (5)
>
>             input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       1.9M     0  1.3M       194M       540k     0        57M 0     0
>       2.1M     0  1.5M       216M       556k     0        58M 0     0
>       1.8M     0  1.3M       192M       553k     0        58M 0     0
>       1.7M     0  1.1M       174M       542k     0        57M 0     0
>       1.9M     0  1.4M       204M       537k     0        56M 0     0
>       1.6M     0  1.1M       171M       550k     0        58M 0     0
>       1.6M     0  1.1M       173M       546k     0        57M 0     0
>       1.7M     0  1.1M       176M       564k     0        59M 0     0
>       2.0M     0  1.5M       212M       543k     0        57M 0     0
>       2.1M     0  1.5M       219M       557k     0        58M 0     0
>       1.9M     0  1.4M       205M       547k     0        57M 0     0
>       1.7M     0  1.2M       179M       553k     0        58M 0     0
>
> # FreeBSD 11.0 - 4 streams to 80 ports from each client (5)
>
>             input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       3.1M     0  1.8M       326M       1.3M     0       134M 0     0
>       2.6M     0  1.5M       269M       1.1M     0       116M 0     0
>       2.7M     0  1.5M       285M       1.2M     0       127M 0     0
>       2.4M     0  1.3M       257M       1.1M     0       119M 0     0
>       2.7M     0  1.5M       287M       1.3M     0       134M 0     0
>       2.5M     0  1.3M       262M       1.2M     0       127M 0     0
>       2.1M     0  1.1M       224M       1.0M     0       108M 0     0
>       2.7M     0  1.4M       285M       1.4M     0       143M 0     0
>       2.6M     0  1.3M       272M       1.3M     0       136M 0     0
>       2.5M     0  1.4M       265M       1.1M     0       120M 0     0
> # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5)
>
>             input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       1.9M     0  849k       195M       1.0M     0       107M 0     0
>       1.9M     0  854k       196M       1.0M     0       106M 0     0
>       1.9M     0  851k       196M       1.0M     0       107M 0     0
>       1.9M     0  851k       196M       1.0M     0       107M 0     0
>       1.9M     0  851k       196M       1.0M     0       107M 0     0
>       1.9M     0  852k       196M       1.0M     0       107M 0     0
>       1.9M     0  847k       195M       1.0M     0       107M 0     0
>       1.9M     0  836k       195M       1.0M     0       107M 0     0
>       1.9M     0  843k       195M       1.0M     0       107M 0     0
> # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5)
>
>            input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       1.1M   259     0       115M       1.1M     0       115M 0     0
>       1.2M   273     0       124M       1.2M     0       124M 0     0
>       1.1M   200     0       112M       1.1M     0       112M 0     0
>       1.2M   290     0       122M       1.2M     0       122M 0     0
>       1.0M   132     0       107M       1.0M     0       107M 0     0
>       1.1M   303     0       118M       1.1M     0       118M 0     0
>       1.1M   278     0       112M       1.1M     0       112M 0     0
>       1.2M   243     0       122M       1.2M     0       122M 0     0
>       1.1M   168     0       112M       1.1M     0       112M 0     0
>       1.1M   161     0       112M       1.1M     0       112M 0     0
> # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client
> (5)
>
>             input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       1.4M    10     0       144M       1.4M     0       144M 0     0
>       1.5M     6     0       159M       1.5M     0       159M 0     0
>       1.4M     5     0       144M       1.4M     0       144M 0     0
>       1.5M    14     0       158M       1.5M     0       158M 0     0
>       1.4M     5     0       151M       1.4M     0       151M 0     0
>       1.4M    10     0       152M       1.4M     0       152M 0     0
>       1.4M    12     0       148M       1.4M     0       148M 0     0
>       1.5M     9     0       155M       1.5M     0       155M 0     0
>       1.4M    23     0       151M       1.4M     0       151M 0     0
>       1.4M    11     0       151M       1.4M     0       151M 0     0
> # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from
> each client (5)
>
>            input        (Total)           output
>    packets  errs idrops      bytes    packets  errs      bytes colls drop=
s
>       1.4M    15     0       145M       1.4M     0       145M 0     0
>       1.5M    18     0       157M       1.5M     0       157M 0     0
>       1.5M    10     0       156M       1.5M     0       156M 0     0
>       1.5M    15     0       154M       1.5M     0       154M 0     0
>       1.4M    13     0       146M       1.4M     0       146M 0     0
>       1.5M    15     0       156M       1.5M     0       156M 0     0
>       1.5M     9     0       155M       1.5M     0       155M 0     0
>       1.5M    13     0       153M       1.5M     0       153M 0     0
>       1.4M    14     0       145M       1.4M     0       145M 0     0
>       1.4M    17     0       151M       1.4M     0       151M 0     0
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAL7QUyNtfVwjo640yb2PW-UzY%2BL17vghwwjkdNLyMXsyz5SQ%2Bg>