Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 21 Nov 2004 16:38:07 -0800 (PST)
From:      Matthew Dillon <dillon@apollo.backplane.com>
To:        Shunsuke SHINOMIYA <shino@fornext.org>
Cc:        Jeremie Le Hen <jeremie@le-hen.org>
Subject:   Re: Re[2]: serious networking (em) performance (ggate and NFS) problem
Message-ID:  <200411220038.iAM0c7JQ052589@apollo.backplane.com>
References:  <20041119185315.C43D.SHINO@fornext.org> <20041121205158.45CE.SHINO@fornext.org>

next in thread | previous in thread | raw e-mail | index | archive | help
: I did simple benchmark at some settings.
:
: I used two boxes which are single Xeon 2.4GHz with on-boarded em.
: I measured a TCP throughput by iperf.
:
: These results show that the throughput of TCP increased if Interrupt
:Moderation is turned OFF. At least, adjusting these parameters affected
:TCP performance. Other appropriate combination of parameter may exist.

    Very interesting, but the only reason you get lower results is simply
    because the TCP window is not big enough.  That's it.

    8000 ints/sec = ~15KB of backlogged traffic.  x 2 (sender, receiver)

    Multiply by two (both the sender's reception of acks and the receiver's
    reception of data) and you get ~30KB.  This is awefully close to the
    default 32.5KB window size that iperf uses.

    Other then window sizing issues I can think of no rational reason why
    throughput would be lower.  Can you?  And, in fact, when I do the same
    tests on DragonFly and play with the interrupt throttle rate I get
    nearly the results I expect.

    * Shuttle Athlon 64 3200+ box, EM card in 32 bit PCI slot 
    * 2 machines connected through a GiGE switch
    * All other hw.em0 delays set to 0 on both sides
    * throttle settings set on both sides
    * -w option set on iperf client AND server for 63.5KB window
    * software interrupt throttling has been turned off for these tests

    throttle	result		result
    freq	(32.5KB win)	(63.5KB win)
		(default)
    --------	--------------	-----------

  maxrate	481 MBit/s	533 MBit/s	(not sure what's going on here)
  120000	518 MBit/s	558 MBit/s	(not sure what's going on here)
  100000	613 MBit/s	667 MBit/s	(not sure what's going on here)
   70000	679 MBit/s	691 MBit/s
   60000	668 MBit/s	694 MBit/s
   50000	678 MBit/s	684 MBit/s
   40000	694 MBit/s	696 MBit/s
   30000	694 MBit/s	696 MBit/s
   20000	698 MBit/s	703 MBit/s
   10000	707 MBit/s	716 MBit/s
    9000	708 MBit/s	716 MBit/s
    8000	710 MBit/s	717 MBit/s	<--- drop off pt 32.5KB win
    7000	683 MBit/s	716 MBit/s
    6000	680 MBit/s	720 MBit/s
    5000	652 MBit/s	718 MBit/s	<--- drop off pt 63.5KB win
    4000	555 Mbit/s	695 MBit/s
    3000	522 MBit/s	533 MBit/s	<--- GiGE throttling likely
    2000	449 MBit/s	384 MBit/s	(256 ring descriptors =
    1000	260 MBit/s	193 MBit/s	  2500 hz minimum)

    Unless you are in a situation where you need to route small packets
    flying around a cluster where low latency is important, it doesn't really
    make any sense to turn off interrupt throttling.  It might make sense 
    to change the default from 8000 to 10000 to handle typical default
    TCP window sizes (at least in a LAN situation), but it certainly should
    not be turned off.

    I got some weird results when I increased the frequency past 100KHz, and
    when I turned throttling off entirely.  I'm not sure why.  Maybe setting
    the ITR register to 0 is a bad idea.  If I set it to 1 (i.e. 3906250 Hz)
    then I get 625 MBit/s.  Setting the ITR to 1 (i.e. 256ns delay) should
    amount to the same thing as setting it to 0 but it doesn't.  Very odd.
    The maximum interrupt rate as reported by systat is only ~46000 ints/sec
    so all the values above 50KHz should read about the same... and they
    do until we hit around 100Khz (10uS delay).  Then everything goes to
    hell in a handbasket.

    Conclusion: 10000 hz would probably be a better default then 8000 hz.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200411220038.iAM0c7JQ052589>