From owner-freebsd-arch@freebsd.org Wed Aug 26 16:36:09 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 164809C3576 for ; Wed, 26 Aug 2015 16:36:09 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9945410C5; Wed, 26 Aug 2015 16:36:08 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by laba3 with SMTP id a3so123359985lab.1; Wed, 26 Aug 2015 09:36:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DbRAPayfsW6+JWlPkgVNl6ni7oZPoWyeQwM97m6DRDo=; b=WiNK64WRwPhG5KFKRATDYkksCeGW3yCvyhym79U+cdHDWScDGSt6laXadvtvBSFOec /fpEgKx0M+GrlIflaFBVunm1pJqi8f1zBdD4fY+gXL+C4TRk58r+aW3D82DvNmcDReJk gH9Mc/FrX0ixQm3gY8SWzNVKNcUkByYlFMaLN8ZLN4dLU5YXmNpygnCNds0MKu3OtiEv VfHbzN4bfjGLzzGBk8c/W0Hh1VYmJoQrdA9Uj7LGcxuuzNVcqPVdndA5j8kZxjE3+bIj PeH04WxEjm9WRNOB0Jr8lReUfS0UIFwT1EBW1PPB7QIX088wVWWc8VoyRK7U8H7f8Qyc eNeA== MIME-Version: 1.0 X-Received: by 10.152.29.6 with SMTP id f6mr30754084lah.85.1440606966581; Wed, 26 Aug 2015 09:36:06 -0700 (PDT) Received: by 10.112.19.1 with HTTP; Wed, 26 Aug 2015 09:36:06 -0700 (PDT) In-Reply-To: <55DDE9B8.4080903@freebsd.org> References: <55DDE9B8.4080903@freebsd.org> Date: Wed, 26 Aug 2015 09:36:06 -0700 Message-ID: Subject: Re: Network card interrupt handling From: Jack Vogel To: Sean Bruno Cc: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 16:36:09 -0000 I recall actually trying something like this once myself Sean, but if memory serves the performance was poor enough that I decided against pursuing it. Still, maybe it deserves further investigation. Jack On Wed, Aug 26, 2015 at 9:30 AM, Sean Bruno wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > We've been diagnosing what appeared to be out of order processing in > the network stack this week only to find out that the network card > driver was shoveling bits to us out of order (em). > > This *seems* to be due to a design choice where the driver is allowed > to assert a "soft interrupt" to the h/w device while real interrupts > are disabled. This allows a fake "em_msix_rx" to be started *while* > "em_handle_que" is running from the taskqueue. We've isolated and > worked around this by setting our processing_limit in the driver to > - -1. This means that *most* packet processing is now handled in the > MSI-X handler instead of being deferred. Some periodic interference > is still detectable via em_local_timer() which causes one of these > "fake" interrupt assertions in the normal, card is *not* hung case. > > Both functions use identical code for a start. Both end up down > inside of em_rxeof() to process packets. Both drop the RX lock prior > to handing the data up the network stack. > > This means that the em_handle_que running from the taskqueue will be > preempted. Dtrace confirms that this allows out of order processing > to occur at times and generates a lot of resets. > > The reason I'm bringing this up on -arch and not on -net is that this > is a common design pattern in some of the Ethernet drivers. We've > done preliminary tests on a patch that moves *all* processing of RX > packets to the rx_task taskqueue, which means that em_handle_que is > now the only path to get packets processed. > > > https://people.freebsd.org/~sbruno/em_interupt_to_taskqueue.diff > > My sense is that this is a slightly "better" method to handle the > packets but removes some immediacy from packet processing since all > processing is deferred. However, all packet processing is now > serialized per queue, which I think is the proper implementation. > > Am I smoking "le dope" here or is this the way forward? > > sean > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQF8BAEBCgBmBQJV3em1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w > ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRCQUFENDYzMkU3MTIxREU4RDIwOTk3REQx > MjAxRUZDQTFFNzI3RTY0AAoJEBIB78oecn5klyYH+wX22JSRYkKMeCJGVSi1dJiM > fcd+DWZVhru2qyUNEfhBSoGEgi7HzXqaBwddy7GK2IRtbKeRlF/oebsII941SIsz > t2f35MoZunw0rWObIEw4WxxkXAajeATDjx87wozVmsZZ40JbmgZ0jKIGXiNie3Is > 04pkXiIOElWqjlLtFlkITUUrOeKsN7kKbwaZAHYeFRdbUgsnxsh7fRvsFucOCgyr > CONHBGWEwu/g50YUruR+rPOHFAA1HD3dQuIoHcTjQx/uX4l5bw+8CFzzMjpw6X9d > G+boH6l1ZZ6U3uZCXOSmkPiXka7Ix8rdbUyrUrJTJrGEB7+U7rF2lSSq8cX+4pk= > =UibL > -----END PGP SIGNATURE----- > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >