Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 19 Oct 2005 23:20:00 +1000
From:      Michael VInce <mv@roq.com>
To:        Robert Watson <rwatson@FreeBSD.org>
Cc:        freebsd-net@freebsd.org, stable@freebsd.org
Subject:   Re: Network performance 6.0 with netperf
Message-ID:  <43564800.3010309@roq.com>
In-Reply-To: <20051014205434.C66245@fledge.watson.org>
References:  <434FABCC.2060709@roq.com> <20051014205434.C66245@fledge.watson.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson wrote:

>
> On Fri, 14 Oct 2005, Michael VInce wrote:
>
>> I been doing some network benchmarking using netperf and just simple 
>> 'fetch' on a new network setup to make sure I am getting the most out 
>> of the router and servers, I thought I would post some results in 
>> case some one can help me with my problems or if others are just 
>> interested to see the results.
>
>
> Until recently (or maybe still), netperf was compiled with -DHISTOGRAM 
> by our port/package, which resulted in a significant performance 
> drop.  I believe that the port maintainer and others have agreed to 
> change it, but I'm not sure if it's been committed yet, or which 
> packages have been rebuilt.  You may want to manually rebuild it to 
> make sure -DHISTOGRAM isn't set.
>
> You may want to try setting net.isr.direct=1 and see what performance 
> impact that has for you.
>
> Robert N M Watson

I reinstalled the netperf to make sure its the latest.

I have also decided to upgrade Server-C (the i386 5.4 box) to 6.0RC1 and 
noticed it gave a large improvement of network performance with a SMP 
kernel.

As with the network setup ( A --- B --- C  ) with server B being the 
gateway, doing a basic 'fetch' from the gateway (B) to the Apache server 
(C) it gives up to 700mbits/sec transfer performance, doing a fetch from 
server A thus going through the gateway gives slower but still decent 
performance of up to 400mbits/sec.

B> fetch -o - > /dev/null http://server-c/file1gig.iso
-                                             100% of 1055 MB   69 MBps 
00m00s


A> fetch -o - > /dev/null http://server-c/file1gig.iso
-                                             100% of 1055 MB   39 MBps 
00m00s

Netperf from the gateway directly to the apache server (C) 916mbits/sec
B> /usr/local/netperf/netperf -l 20 -H server-C -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 916.50

Netperf from the client machine through the gateway to the apache server 
(C) 315mbits/sec
A> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 315.89

Client to gateway netperf test shows the direct connection between these 
machines is fast. 912mbits/sec
A> /usr/local/netperf/netperf -l 30 -H server-B -t TCP_STREAM -i 10,2 -I 
99,5 -- -m 4096 -s 57344 -S 5734
Elapsed Throughput - 10^6bits/sec: 912.11

The strange thing now is in my last post I was able to get faster speeds 
from server A to C with 'fetch' tests on non-smp kernels and slower 
speeds with netperf tests. Now I get speeds a bit slower with fetch 
tests but faster netperf speed tests with or without SMP on server-C.

I was going to test with 'net.isr.dispatch' but the sysctl doesn't 
appear to exist, doing this returns nothing.
'sysctl -a | grep 'net.isr.dispatch'

I also tried polling but its also like that doesn't exist either.
ifconfig em3 inet 192.168.1.1 netmask 255.255.255.224 polling
ifconfig: polling: Invalid argument

When doing netperf tests there was high interrupt usage.
CPU states:  0.7% user,  0.0% nice, 13.5% system, 70.0% interrupt, 15.7% 
idle

Also the server B is using its last 2 gigabit ethernet ports which are 
listed from pciconf -lv as '82547EI Gigabit Ethernet Controller'
While the first 2 are listed as 'PRO/1000 P'
Does any one know if the PRO/1000P would be better?

em0@pci5:4:0:   class=0x020000 card=0x118a8086 chip=0x108a8086 rev=0x03 
hdr=0x00
    vendor   = 'Intel Corporation'
    device   = 'PRO/1000 P'

em3@pci9:8:0:   class=0x020000 card=0x016d1028 chip=0x10768086 rev=0x05 
hdr=0x00
    vendor   = 'Intel Corporation'
    device   = '82547EI Gigabit Ethernet Controller'

Cheers,
Mike

>
>>
>> The network is currently like this, where machines A and B are the 
>> Dell 1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM 
>> on it) and server B is the gateway and A is acting as a client for 
>> fetch and netperf tests.
>> A --- B --- C
>> The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is 
>> running 5.4-stable i386 from Oct 12
>>
>> My main problem is that if I compile SMP into the machine C 
>> (5.4stable) the network speed goes down to a range between 
>> 6mbytes/sec to 15mbytes/sec on SMP.
>> If I use GENERIC kernel the performance goes up to what I have show 
>> below which is around 65megabytes/sec for a 'fetch' get test from 
>> Apache server and 933mbits/sec for netperf.
>> Does any know why why network performance would be so bad on SMP?
>>
>> Does any one think that if I upgrade the i386 SMP server to 6.0RC1 
>> the SMP network performance would improve? This server will be 
>> running java so I need it to be stable and is the the reason I am 
>> using i386 and Java 1.4
>>
>> I am happy with performance of direct machine to machine (non SMP) 
>> which is pretty much full 1gigabit/sec speeds.
>> Going through the gateway server-B seems to drop its speed down a bit 
>> for in and out direction tcp speed tests using netperf I get around 
>> 266mbits/sec from server A through gateway Server-B to server-C which 
>> is quite adequate for the link I currently have for it.
>>
>> Doing a 'fetch' get for a 1gig file from the Apache server gives good 
>> speeds of close to 600mbits/sec but netperf shows its weakness with 
>> 266mbits/sec.
>> This is as fast as I need it to be but does any one know the weak 
>> points on the router gateway to make it faster? Is this the 
>> performance I should expect for FreeBSD as a router with gigabit ethers?
>>
>> I have seen 'net.inet.ip.fastforwarding' in some peoples router 
>> setups on the list but nothing about what it does or what it can affect.
>> I haven't done any testing with polling yet but if I can get over 
>> 900mbits/sec on the interfaces does polling help with passing packets 
>> from one interface to the other?
>> All machines have PF running other then that they don't really have 
>> any sysctls or special kernel options.
>>
>> Here are some speed benchmarks using netperf and 'fetch' gets.
>>
>> Server A to server C with server C using SMP kernel and just GENERIC 
>> kernel further below
>>
>> B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 
>> -I 99,5 -- -m 4096 -s 57344 -S 57344
>> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 57344  57344   4096    10.06     155.99
>> tank# fetch -o - > /dev/null http://server-C/file1gig.iso
>> -                                             100% of 1055 MB   13 
>> MBps 00m00s
>>
>> ##### Using generic non SMP kernel
>> Server A to server C with server C using GENERIC kernel.
>> A# fetch -o - > /dev/null http://server-C/file1gig.iso
>> -                                             100% of 1055 MB   59 
>> MBps 00m00s
>>
>> A# ./tcp_stream_script server-C
>>
>> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 
>> 99,5 -- -m 4096 -s 57344 -S 57344
>>
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 57344  57344   4096    60.43     266.92
>>
>> ------------------------------------
>> ###############################################
>> Connecting from server-A to B (gateway)
>> A# ./tcp_stream_script server-B
>>
>> ------------------------------------
>>
>> /usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I 
>> 99,5 -- -m 4096 -s 57344 -S 57344
>>
>> TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 57344  57344   4096    61.80     926.82
>>
>> ------------------------------------
>> ##########################################
>> Connecting from server B (gateway) to server C
>> Fetch and Apache2 test
>> B# fetch -o - > /dev/null http://server-C/file1gig.iso
>> -                                             100% of 1055 MB   74 
>> MBps 00m00s
>>
>> Netperf test
>> B# /usr/local/netperf/tcp_stream_script server-C
>>
>> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 
>> 99,5 -- -m 4096 -s 57344 -S 57344
>>
>> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 57344  57344   4096    62.20     933.94
>>
>> ------------------------------------
>>
>> Cheers,
>> Mike
>>
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43564800.3010309>