Date: Thu, 2 Oct 2014 01:07:53 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Adrian Chadd <adrian@freebsd.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, "Alexander V. Chernikov" <melifaro@freebsd.org>, Elof Ofel <elofu17@hotmail.com>, jfvogel@gmail.com Subject: bug confirmed: Re: individual queue blocking entire rx unit on ixgbe (Re: How do I balance bandwidth over several virtual NICs?) Message-ID: <20141001230753.GA13187@onelab2.iet.unipi.it> In-Reply-To: <20141001203223.GA12122@onelab2.iet.unipi.it> References: <CA%2BhQ2%2BhYY7AWE7jJWAg8uYER2LRSPz8TcVPuamsRBLj_WQyakw@mail.gmail.com> <CAJ-Vmo=xiZtKkQEV30Y_c7a6uWD6PbuqzmSSDG12J=w8Z45biA@mail.gmail.com> <20141001203223.GA12122@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 01, 2014 at 10:32:23PM +0200, Luigi Rizzo wrote:
> On Wed, Oct 01, 2014 at 10:37:35AM -0700, Adrian Chadd wrote:
> > Hi,
> >
> > Try this on -HEAD. There were some recent fixes to ixgbe that haven't
> > been RFCed.
>
> I don't have a way to test this on HEAD.
>
> Do you know if there is any change that could be related ?
>
> On 10.1 beta 3 I noticed is that when I open a single queue on an ixgbe
> (and with incoming traffic), the rx unit stalls no matter what the
> previous state of dev.ix.*.fc (i.e. the DROP_EN bits) or QDE are.
>
> In this state, toggling DROP_EN has no effect, whereas something that seems
> effective is setting the QDE bit(s) in the PFQDE register for all queues,
> _after_ i open the device.
>
> >From the above my take is the following:
>
> - on NIC reset, the SRRCTL register starts at 0 including DROP_EN;
> same goes for PFQDE.QDE
>
> - setting SRRCTL.DROP_EN must happen before the receive unit is started;
>
> - conversely, toggling PFQDE.QDE has effect even when the receive
> unit has started
>
> - sysctl dev.ix.*.fc sets/clear DROP_EN but at the wrong time
> (the right time seems to be the window between reset and start)
>
> I am going to run more tests to figure out.
ok i can confirm that SRRCTL.DROP_EN behaves as i described above,
and so it should be set in ixgbe_initialize_receive_units() .
The patch below shows that DROP_EN is cleared when the NIC is reset,
and sets it according to the flow control setting. To replicate
exactly the logic for flow control we might even try to check
whether there is more than 1 queue.
Also the previous code that modifies DROP_EN can be probably
removed because it is ineffective.
cheers
luigi
--- /home/luigi/FreeBSD/R10/sys/dev/ixgbe/ixgbe.c 2014-08-21 01:29:41.0000
00000 +0200
+++ ixgbe.c 2014-10-02 00:38:00.000000000 +0200
@@ -4187,6 +4192,17 @@ ixgbe_initialize_receive_units(struct ad
/* Set up the SRRCTL register */
srrctl = IXGBE_READ_REG(hw, IXGBE_SRRCTL(i));
+{
+ printf("--- queue %d DROP_EN is %d\n",
+ i, (srrctl & IXGBE_SRRCTL_DROP_EN) ? 1 : 0);
+ if (adapter->hw.fc.requested_mode == ixgbe_fc_none) {
+ srrctl |= IXGBE_SRRCTL_DROP_EN;
+ } else {
+ srrctl &= ~IXGBE_SRRCTL_DROP_EN;
+ }
+ printf("--- queue %d DROP_EN becomes %d\n",
+ i, (srrctl & IXGBE_SRRCTL_DROP_EN) ? 1 : 0);
+}
srrctl &= ~IXGBE_SRRCTL_BSIZEHDR_MASK;
srrctl &= ~IXGBE_SRRCTL_BSIZEPKT_MASK;
srrctl |= bufsz;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141001230753.GA13187>