Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 8 Oct 2005 14:39:26 +0100 (BST)
From:      Robert Watson <rwatson@FreeBSD.org>
To:        arch@FreeBSD.org
Subject:   Call for performance evaluation: net.isr.direct (fwd)
Message-ID:  <20051008143854.B84936@fledge.watson.org>

next in thread | raw e-mail | index | archive | help

FYI, as this is a general architectural issue.  Please follow up to 
performance@/net@.

Thanks,

Robert N M Watson

---------- Forwarded message ----------
Date: Wed, 5 Oct 2005 17:12:14 +0100 (BST)
From: Robert Watson <rwatson@FreeBSD.org>
To: performance@FreeBSD.org
Cc: net@FreeBSD.org
Subject: Call for performance evaluation: net.isr.direct


In 2003, Jonathan Lemon added initial support for direct dispatch of netisr 
handlers from the calling thread, as part of his DARPA/NAI Labs contract in the 
DARPA CHATS research program.  Over the last two years since then, Sam Leffler 
and I have worked to refine this implementation, removing a number of ordering 
related issues, opportunities for excessive parallelism, recursion issues, and 
testing with a broad range of network components.  There has also been a 
significant effort to complete MPSAFE locking work throughout the network 
stack.  Combined with the earlier move to ithreads and a functional direct 
dispatch ("process to completion" implementation), there are a number of 
exciting possible benefits.

- Possible parallelism by packet source -- ithreads can dispatch
   simultaenously into the higher level network stack layers.  Since
   ithreads can execute in parallel on different CPU, so can code they
   invoke directly.

- Elimination of context switches in the network receive path -- rather
   than context switching to the netisr thread from the ithread, we can now
   directly execute netisr code from the ithread.

- A CPU-bound netisr thread on a multi-processor system will no longer
   rate limit traffic to the available resources on one CPU.

- Eliminating the additional queueing in the handoff reduces the
   opportunity for queues to overfill as a result of scheduling delays.

There are, however, some possible downsides and/or trade-offs:

- Higher level network processing will now compete with the interrupt
   handler for CPU resources available to the ithread.  This means less
   time for the interrupt code to execute in the thread if the thread is
   CPU-bound.

- Lower levels of parallelism between portions of the inbound packet
   processing path.  Without direct dispatch, there is possible parallelism
   between receive network driver execution and higher level stack layers,
   whereas with direct dispatch they can no longer execute in parallel.

- Re-queued packets from tunnel and encapsulation processing will now
   require a context switch to process, since they will be processed in the
   netisr proper rather than in the ithread, whereas before the netisr
   thread would pick them up immediately after completing the current
   processing without a context switch.

- Code that previously ran in the SWI at a SWI priority now runs in the
   ithread at an ithread priority, elevating the general priority at which
   network processing takes place.

And there are a few mixed things, that can offer good and bad elements:

- Less queueing takes place in the network stack in in-bound processing:
   packets are taken directly from the driver and processed to completion
   one by one, rather than queued for batch processing.  Packets will be
   dropped before the link layer, rather than on the boundary between the
   link and protocol layers.  This is good in that we invest less work in
   packets we were going to drop anyway, but bad in that less queueing
   means less room for scheduling delays.

In previous FreeBSD releases, such as several 5.x series releases, 
net.isr.enable could not be turned on by default because there was insufficient 
synchronization in the network stack.  As of 5.5 and 6.0, I believe there is 
sufficient synchronization, especially given that we force non-MPSAFE protocol 
handlers to run in the netisr without direct dispatch.  As such, there has been 
a gradual conversation going on about making direct dispatch the default 
behavior in the 7.x development series, and more publically documenting and 
supporting the use of direct dispatch in the 6.x release engineering series.

Obviously, this is about two things: performance, and stability.  Many of us 
have been running with direct dispatch on by default for quite some time, so it 
passes some of the basic "does it run" tests.  However, since it significantly 
increases the opportunity for parallelism in the receive path of the network 
stack, it likely will trigger otherwise latent or infrequent races and bugs to 
occur more frequently.  The second aspect is performance: many results suggest 
that direct dispatch has a significant performance benefit.  However, 
evaluating the impact on a broad range of results is required in order for us 
to go ahead with what is effectively a significant architectural change in how 
we perform network stack processing.

To give you a sense of some of the performance effect I've measured recently, 
using the netperf measurement tool (with -DHISTOGRAM removed from the FreeBSD 
port build), here are some results.  In each case, I've put parenthesis around 
host or router to indicate which is the host where the configuration change is 
being tested.  These tests were performed using dual Xeon systems, and using 
back-to-back gigabit ethernet cards and the if_em driver:

TCP round trip benchmark (TCP_RR), host-(host):

7.x UP: 0.9% performance improvement
7.x SMP: 0.7% performance improvement

TCP round trip benchmark (TCP_RR), host-(router)-host:

7.x UP: 2.4% performance improvement
7.x SMP: 2.9% performance improvement

UDP round trip benchmark (UDP_RR), host-(host):

7.x UP: 0.7% performance improvement
7.x SMP: 0.6% performance improvement

UDP round trip benchmark (UDP_RR), host-(router)-host:

7.x UP: 2.2% performance improvement
7.x SMP: 3.0% performance improvement

TCP stream banchmark (TCP_STREAM), host-(host):

7.x UP: 0.8% performance improvement
7.x SMP: 1.8% performance improvement

TCP stream benchmark (TCP_STREAM), host-(router)-host:

7.x UP: 13.6% performance improvement
7.x SMP: 15.7% performance improvement

UDP stream benchmark (UDP_STREAM), host-(host):

7.x UP: none
7.x SMP: none

UDP stream benchmark (UDP_STREAM), host-(router)-host:

7.x UP: none
7.x SMP: none

TCP connect benchmark (src/tools/tools/netrate/tcpconnect)

7.x UP: 7.90383% +/- 0.553773%
7.x SMP: 12.2391% +/- 0.500561%

So in some cases, the impact is negligible -- in other places, it is quite 
significant.  So far, I've not measured a case where performance has gotten 
worse, but that's probably because I've only been measuring a limited number of 
cases, and with a fairly limited scope of configurations, especially given that 
the hardware I have is pushing the limits of what the wire supports, so minor 
changes in latency are possible, but not large changes in throughput.

So other than a summary of the status quo, this is also a call to action. I 
would like to get more widespread benchmarking of the impact of direct dispatch 
on network-related workloads.  This means a variety of things:

(1) Performance of low level network services, such as routing, bridging,
     and filtering.

(2) Performance of high level application servces, such as web and
     database.

(3) Performance of integrated kernel network services, such as the NFS
     client and server.

(4) Performance of user space distributed file systems, such as Samba and
     AFS.

All you need to do to switch to direct dispatch mode is set the sysctl or 
tunable "net.isr.dispatch" to 1.  To disable it again, remove the setting, or 
set it to 0.  It can be modified at run-time, although during the transition 
from one mode to the other, there may be a small quantity of packet 
misordering, so benchmarking over the transition is discouraged.
FYI: as of 6.0-RC1 and recent 7.0, net.isr.dispatch is the name of the 
variable.  In earlier releases, the name of this variable was net.isr.enable.

Some important details:

- Only non-local protocol traffic is affected: loopback traffic still goes
   via the netisr to avoid issues of recursion and lock order.

- In the general case, only in-bound traffic is directly affected by this
   change.  As such, send-only benchmarks may reveal little change.  They
   are still interesting, however.

- However, the send path is indirectly affected due to changes in
   scheduling, workload, interrupt handling, and so on.

- Because network benchmarks, especially micro-benchmarks, are especially
   sensitive to minor perturbations, I highly recommend running in a
   minimal multi-user or ideally single-user environment, and suggest
   isolating undesired sources of network traffic from segments where
   testing is occuring.  For macro-benchmarks this can be less important,
   but should be paid attention to.

- Please make sure debugging features are turned off when running tests --
   especially WITNESS, INVARIANTS, INVARIANT_SUPPORT, and user space malloc
   debugging.  These can have a significant impact on performance, both
   potentially overshadowing changes, and in some cases, actually reversing
   results (due to higher overhead under locks, for example).

- Do not use net.isr.enable in the 5.x line unless you know what you are
   doing.  While it is reasonably safe with 5.4 forwards, it is not a
   supported configuration, and may cause stability issues with specific
   workloads.

- What we're particularly interested in is a statistically meaningful
   comparison of the "before" and "after" case.  When doing measurements, I
   like to run 10-12 samples, and usually discard the first one or two,
   depending on the details of the benchmark.  I'll then use
   src/tools/tools/ministat to compare the data sets.  Running a number of
   samples is quite important, because the variance in many tests can be
   significant, and if the two sample sets overlap, you can quite easily
   draw the entirely wrong conclusion about the results from a small number
   of measurements in a sample.

Assuming you have a fixed width font, typicaly output from ministat looks 
something like the following and may be human readable:

x 7SMP/tcpconnect_queue
+ 7SMP/tcpconnect_direct
+--------------------------------------------------------------------------+
|x xx                                                                +    +|
|xxxxx  xx                                                       ++ +++++ +|
||__A__|                                                          |___A__| |
+--------------------------------------------------------------------------+
     N           Min           Max        Median           Avg        Stddev
x  10          5425          5503          5460        5456.3     26.284977
+  10          6074          6169          6126        6124.1     31.606785
Difference at 95.0% confidence
         667.8 +/- 27.3121
         12.2391% +/- 0.500561%
         (Student's t, pooled s = 29.0679)

Of particular interest is if changing to direct dispatch hurts performance in 
your environment, and understanding why that is.

Thanks,

Robert N M Watson
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051008143854.B84936>