Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 03 Jul 2008 02:47:45 -0400
From:      Paul <paul@gtcomm.net>
To:        Ingo Flaschberger <if@xip.at>
Cc:        FreeBSD Net <freebsd-net@freebsd.org>
Subject:   Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]
Message-ID:  <486C7611.9030905@gtcomm.net>
In-Reply-To: <alpine.LFD.1.10.0807021052041.557@filebunker.xip.at>
References:  <4867420D.7090406@gtcomm.net>	<200806301944.m5UJifJD081781@lava.sentex.ca>	<20080701004346.GA3898@stlux503.dsto.defence.gov.au>	<alpine.LFD.1.10.0807010257570.19444@filebunker.xip.at>	<20080701010716.GF3898@stlux503.dsto.defence.gov.au>	<alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at>	<486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net>	<ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com>	<20080701033117.GH83626@cdnetworks.co.kr>	<ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com>	<4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net>	<486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net>	<486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <alpine.LFD.1.10.0807021052041.557@filebunker.xip.at>

next in thread | previous in thread | raw e-mail | index | archive | help
Preliminary 32 bit results...
When I started out it looked like 32 bit was worse than 64 bit, but it's 
just the timers are different.
For instance, 4000 hz in 64 bit gives better results than 4000hz in 32 bit.
Low HZ gives better result with polling on in 32 bit

Bottom line, so far I'm not able to get any better performance out of 32 
bit at all.  In fact I think it might be even a tad slower. I didn't see 
as high of bursts like I did on 64 bit so far but I'm still testing.

Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I can 
see if it scales directly with cpu speed or what happens.

I did another SMP test with an interesting results. I took one of the 
cpus out of the machine, so it was just left with a single 2212 (dual core)
and it performed better.  Less contention I suppose?




some results:
kern.hz=4000
hw.em.rxd=512
hw.em.txd=512

polling on, idle polling on (only way I can get a reliable netstat output)

           input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    681961 117612   42281586          1     0        226     0
    655095 83418   40615892          2     0        220     0
    683881 93559   42400626          1     0        178     0
    683637 90452   42385498          1     0        178     0
    683345 87471   42367394          1     0        178     0
    682737 81483   42329696          2     0        220     0
    683154 95413   42355552          1     0        178     0
    684556 111013   42442476          1     0        178     0
    684365 110960   42430634          1     0        178     0
    679089 116440   42103518          3     0        534     0
    684328 122713   42428340          1     0        178     0
    684852 121387   42460828          1     0        178     0
    685358 113256   42492200          1     0        178     0
    685060 123110   42473724          1     0        178     0
    684463 118335   42436710          1     0        178     0
    677182 127788   41985300          2     0        356     0
    685920 126144   42527044          1     0        178     0
    684946 107034   42466656          1     0        178     0


(reboot)
kern.hz=1000
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    679611 97394   42136046          5     0        762     0
    663939 104714   41164254          5     0       1322     0
    685538 91102   42503412          4     0        536     0
    676704 94629   41955668          2     0        404     0
    685323 115060   42490030          1     0        178     0
    675954 105506   41909164          2     0        356     0
    655321 92118   40629906          1     0        178     0
    686826 85674   42583228          2     0        356     0
    686378 89983   42555440          1     0        178     0
    685539 80180   42503422          1     0        178     0
    686704 88626   42575652          1     0        178     0
    686567 88596   42567158          1     0        178     0
    687031 82640   42595936          3     0        398     0
sysctl -w kern.polling.each_burst=50
kern.polling.each_burst: 256 -> 50
[root@ircrouter ~]# netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    693036 39992   42968315          3     0        400     0
    695538 58189   43123360          1     0        178     0
    692670 62765   42945544          1     0        178     0
    693219 60755   42979580          2     0        220     0
    692637 64761   42943498          1    

sysctl -w kern.polling.each_burst=33
kern.polling.each_burst: 50 -> 33
[root@ircrouter ~]# netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    690530 63359   42812868          1     0        226     0
    689748 57670   42764380          1     0        178     0
    690489 57874   42810322          1     0        178     0
    689655 60606   42758614          1     0        178     0
^C
[root@ircrouter ~]# sysctl -w kern.polling.each_burst=3
kern.polling.each_burst: 33 -> 3
[root@ircrouter ~]# netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    612234 110896   37958512          1     0        226     0
    614391 112506   38092246          1     0        178     0
^C
[root@ircrouter ~]# sysctl -w kern.polling.each_burst=800
kern.polling.each_burst: 3 -> 800
[root@ircrouter ~]# netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    668057 76496   41419538          1     0        226     0
    667689 88674   41396720          2     0        220     0
    670526 106654   41572616          1     0        178     0
    667326 97832   41374216          1     0        178     0
^C
[root@ircrouter ~]# sysctl -w kern.polling.each_burst=66
kern.polling.each_burst: 800 -> 66
[root@ircrouter ~]# netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    690164 89290   42790172          1     0        226     0
    688886 74360   42710936          1     0        178     0
    674079 77027   41792902          1     0        178     0

kern.hz=2000
           input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    699116 238016   43345196          1     0        178     0
    698263 225244   43292310          1     0        290     0
    697246 222395   43229256          1     0        178     0
    696749 207766   43198442          1     0        178     0
    697304 217384   43232852          1     0        178     0
    696401 209901   43176866          1     0        178     0
    696508 207757   43183500          1     0        178     0
^C

hz=2000 with 1024/1024 descriptors
          input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    670315 235780   41559534          1     0        226     0
    683218 225838   42359520          1     0        178     0
    682998 242551   42345880          1     0        178     0
    681777 239649   42270178          1     0        178     0


hz=1000 with 256/256 descriptors

netstat -w1 -I em0
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    740584 160355   45916212          2     0          0     0
    746027 165198   46253678          1     0        178     0
    746068 165921   46256220          1     0        178     0
    746505 167527   46283314          1     0        178     0
    743902 175019   46121928          1     0        178     0
    746130 179795   46260064          1     0        178     0
    744457 166448   46156338          1     0        178     0
    746169 176137   46262482          1     0        178     0

hz=667 with 256/256
           input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    742614 91687   46042072          1     0        226     0
    739746 85695   45864256          1     0        178     0
    733723 85162   45490840          3     0        398     0
    737561 102207   45728786          1     0        178     0
    739618 127597   45856320          1     0        178     0
^C

Hrm finally same pps as 64 bit.....
Now I wonder what happens if I go back to the 64 bit and try 1000 
256/256 ?? I don't think I tried that one..
Guess another reinstall :>
Installing 64 bit.. (again) Just to be sure..





Paul







Ingo Flaschberger wrote:
> Dear Paul,
>
>> SMP DISABLED on my Opteron 2212  (ULE, Preemption on)
>> Yields ~750kpps in em0 and out em1  (one direction)
>> I am miffed why this yields more pps than
>> a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 
>> 3 incoming connections so 3 taskq threads
>
> because less locking, less synchronisation, ....
>
>> I would be willing to set up test equipment (several servers plugged 
>> into a switch) with ipkvm and power port access
>> if someone or a group of people want to figure out ways to improve 
>> the routing process, ipfw, and lagg.
>>
>> Maximum PPS with one ipfw rule on UP:
>> tops out about 570Kpps.. almost 200kpps lower ? (frown)
>
> can you post the rule here?
>
>> I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's 
>> in here and see how that scales, using UP same kernel etc I have now.
>
> really, please try 32bit and 1 cpu.
>
> Kind regards,
>     Ingo Flaschberger
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?486C7611.9030905>