From owner-freebsd-current@FreeBSD.ORG Fri Jul 27 06:15:22 2012 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63F7D106566C; Fri, 27 Jul 2012 06:15:22 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 209768FC0C; Fri, 27 Jul 2012 06:15:21 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 8CE267300A; Fri, 27 Jul 2012 08:35:14 +0200 (CEST) Date: Fri, 27 Jul 2012 08:35:14 +0200 From: Luigi Rizzo To: Adrian Chadd Message-ID: <20120727063514.GA49988@onelab2.iet.unipi.it> References: <20120724202019.GA22927@onelab2.iet.unipi.it> <20120725151403.GA33640@onelab2.iet.unipi.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org Subject: Re: RFC: use EM_LEGACY_IRQ in if_lem.c ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2012 06:15:22 -0000 On Thu, Jul 26, 2012 at 10:52:08PM -0700, Adrian Chadd wrote: > On 25 July 2012 08:14, Luigi Rizzo wrote: > > >> I suggest doing some digging to understand why. I bet we all know the > >> answer, but it would be nice to have it documented and investigated. I > >> bet em(4) isn't the only device that would benefit from this? > > > > I am not so sure i know the answer on bare iron (and my take is that the > > difference is more or less irrelevant there), but in the virtualized case > > the improvement is almost surely because the code used in FAST_INTR > > has a couple of MMIO accesses to disable/enable interrupts on the > > card while the taskqueue runs. These are expensive in a VM > > (such accesses cost ~10K cycles each, even with hw support) > > Hm, really? Doing these register accesses to a virtualised em NIC in a > VM is that expensive, or is there something else going on I don't > understand? access to emulated registers is expensive because you need to exit from the hardware VM before you can run the emulation code, and the exit is incredibly expensive as it needs to save a ton of state. Things are marginally better in qemu, but there everything is more expensive so you still see a measurable performance gain if you save the accesses. There is a recent paper/talk at usenix that discusses briefly the problem https://www.usenix.org/conference/usenixfederatedconferencesweek/software-techniques-avoiding-hardware-virtualization-exits The ~10k cycles per access were measured in kvm on recent i5 and i7 processors. For qemu, the change moves the packet rate from 7.5 to 8.3 Kpps, if you do the math the difference is about 13us per packet so we are in a similar ballpark. (the numbers are low because the default emulation does not support interrupt mitigation, i have patches for that, see the qemu-devel mailing list http://lists.nongnu.org/archive/html/qemu-devel/2012-07/msg03195.html which bring the rate up to some 50kpps. cheers luigi