Date: Thu, 27 Dec 2012 10:46:51 -0800 From: Jack Vogel <jfvogel@gmail.com> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: emulation@freebsd.org, current@freebsd.org Subject: Re: [RFC] proposed 'lem' patch to improve behaviour under emulation Message-ID: <CAFOYbckd4G76Pm05pJKW_td-C6JpXtc2_8H%2BJc_FvSys1X62gw@mail.gmail.com> In-Reply-To: <20121227094649.GA48891@onelab2.iet.unipi.it> References: <20121227094649.GA48891@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
LOL, it's ironic, my intention in creating lem was to isolate the old pre-PCIE driver from active changes so as to assure it's stability... but virtualization comes around to bit me in the butt :) I guess I'm agreeable in principle with what you're doing Luigi, but can you do me a favor and hold off until I'm technically back from vacation (after the new year) and let me review the code then? Thanks, Jack On Thu, Dec 27, 2012 at 1:46 AM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > This patch implements two features for the 'lem' driver that > greatly improve the throughput under proper hypervisor. > This is joint work with Vincenzo Maffione and Giuseppe Lettieri, > I am posting it here for review, will then commit it > if there are no objections. > > The first change is to implement a sysctl to access the 'itr' > interrupt moderation register for the devices supported by this > driver. It is little more than adding a struct into the device > descriptor, and one line to create the dynamic sysctl entry, same > as it is done for the other mitigation registers. > > The second change is more interesting and has huge benefits on througput. > > Under virtualization, "VM exits" (which happen every time there is > an access to a register of the emulated peripheral) are extremely > expensive. In the tx path of the 'lem' driver, there is a write > to the TDT register on every packet sent. > > The patch we propose, if enabled through a sysctl (defaults off, > so no change from current behaviour) defers writes to the TDT > register when there is a pending transmit interrupt. > This means that, together with proper emulation of interrupt > mitigation on the hypervisor side, the number of VM exits > is dramatically reduced. To give you an idea, on a modern > system with qemu-kvm and companion patches, UDP throughput is > > KVM QEMU > standard KVM, standard driver 20 Kpps 6.3 Kpps > modified KVM, standard driver 37 Kpps 28 Kpps > modified KVM, modified driver 200 Kpps 34 Kpps > > As you can see, on kvm this change gives a 5x speedup to the tx path, > which combines nicely with the 2x speedup that comes from supporting > interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) > the benefits are much lower, as the guest becomes too slow. > > Patch follows. It would be good if people with real hardware > using the 'lem' driver could test it to make sure it does no > harm on their devices (in any case the sysctl variable > dev.em.0.mit_enable must be set to explicitly enable it > at runtime). > > (for those curious to test it under kvm, i am also attaching a > patch that you need to apply to qemu in order to exploit the > effect of interrupt mitigation; it is a followup of a similar > patch i posted in july to the qemu mailing list, and i will > post it the update there as well, shortly. Unfortunately > we do not have kvm on freebsd..) > > cheers > luigi >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbckd4G76Pm05pJKW_td-C6JpXtc2_8H%2BJc_FvSys1X62gw>