From owner-freebsd-net@FreeBSD.ORG Wed Oct 1 23:02:51 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8FEFF984; Wed, 1 Oct 2014 23:02:51 +0000 (UTC) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF67BC2; Wed, 1 Oct 2014 23:02:51 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id F35FF73027; Thu, 2 Oct 2014 01:07:53 +0200 (CEST) Date: Thu, 2 Oct 2014 01:07:53 +0200 From: Luigi Rizzo To: Adrian Chadd , "freebsd-net@freebsd.org" , "Alexander V. Chernikov" , Elof Ofel , jfvogel@gmail.com Subject: bug confirmed: Re: individual queue blocking entire rx unit on ixgbe (Re: How do I balance bandwidth over several virtual NICs?) Message-ID: <20141001230753.GA13187@onelab2.iet.unipi.it> References: <20141001203223.GA12122@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141001203223.GA12122@onelab2.iet.unipi.it> User-Agent: Mutt/1.5.20 (2009-06-14) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Oct 2014 23:02:51 -0000 On Wed, Oct 01, 2014 at 10:32:23PM +0200, Luigi Rizzo wrote: > On Wed, Oct 01, 2014 at 10:37:35AM -0700, Adrian Chadd wrote: > > Hi, > > > > Try this on -HEAD. There were some recent fixes to ixgbe that haven't > > been RFCed. > > I don't have a way to test this on HEAD. > > Do you know if there is any change that could be related ? > > On 10.1 beta 3 I noticed is that when I open a single queue on an ixgbe > (and with incoming traffic), the rx unit stalls no matter what the > previous state of dev.ix.*.fc (i.e. the DROP_EN bits) or QDE are. > > In this state, toggling DROP_EN has no effect, whereas something that seems > effective is setting the QDE bit(s) in the PFQDE register for all queues, > _after_ i open the device. > > >From the above my take is the following: > > - on NIC reset, the SRRCTL register starts at 0 including DROP_EN; > same goes for PFQDE.QDE > > - setting SRRCTL.DROP_EN must happen before the receive unit is started; > > - conversely, toggling PFQDE.QDE has effect even when the receive > unit has started > > - sysctl dev.ix.*.fc sets/clear DROP_EN but at the wrong time > (the right time seems to be the window between reset and start) > > I am going to run more tests to figure out. ok i can confirm that SRRCTL.DROP_EN behaves as i described above, and so it should be set in ixgbe_initialize_receive_units() . The patch below shows that DROP_EN is cleared when the NIC is reset, and sets it according to the flow control setting. To replicate exactly the logic for flow control we might even try to check whether there is more than 1 queue. Also the previous code that modifies DROP_EN can be probably removed because it is ineffective. cheers luigi --- /home/luigi/FreeBSD/R10/sys/dev/ixgbe/ixgbe.c 2014-08-21 01:29:41.0000 00000 +0200 +++ ixgbe.c 2014-10-02 00:38:00.000000000 +0200 @@ -4187,6 +4192,17 @@ ixgbe_initialize_receive_units(struct ad /* Set up the SRRCTL register */ srrctl = IXGBE_READ_REG(hw, IXGBE_SRRCTL(i)); +{ + printf("--- queue %d DROP_EN is %d\n", + i, (srrctl & IXGBE_SRRCTL_DROP_EN) ? 1 : 0); + if (adapter->hw.fc.requested_mode == ixgbe_fc_none) { + srrctl |= IXGBE_SRRCTL_DROP_EN; + } else { + srrctl &= ~IXGBE_SRRCTL_DROP_EN; + } + printf("--- queue %d DROP_EN becomes %d\n", + i, (srrctl & IXGBE_SRRCTL_DROP_EN) ? 1 : 0); +} srrctl &= ~IXGBE_SRRCTL_BSIZEHDR_MASK; srrctl &= ~IXGBE_SRRCTL_BSIZEPKT_MASK; srrctl |= bufsz;