Date: Sat, 28 Mar 2015 14:16:08 +0100 From: =?windows-1252?Q?Roger_Pau_Monn=E9?= <roger.pau@citrix.com> To: Andrew Daugherity <adaugherity@tamu.edu>, "freebsd-xen@freebsd.org" <freebsd-xen@freebsd.org> Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 Message-ID: <5516A998.10206@citrix.com> In-Reply-To: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, El 28/03/15 a les 0.13, Andrew Daugherity ha escrit: > Summary: FreeBSD 10.1/amd64 under Xen 4.2.5 is much slower than FreeBSD 9.3 on the same environment, especially at fork() > > > I recently installed a FreeBSD-10.1 VM under Xen, and was pleased to see the XENHVM stuff is now integrated into GENERIC. However, the system seemed a little slow and lacking in "snappiness" -- the first fetch/extraction of portsnap was particularly bad, taking at least 20 minutes. It had been a while since I'd done that (as opposed to 'portsnap fetch update') so I wasn't sure how abnormal that was, but then I noticed building stuff from ports, especially stuff using libtool, like security/sssd, was extremely slow compared to physical hardware, so I tested a 9.3 VM, which was much faster. > > Importantly, it was not a typical case of a slow/overloaded CPU but more like slow context switching/forking. I would see high (40%) system CPU percentage but low user, and usually the process at the top of the list was sh. It would take a long time between compiling files but when cc finally ran it was quite fast, compiling each file in a second or two. The system was not swapping and iostat (also xentop on the host) showed minimal I/O load. > > Tracing the sh process (which was libtool-related) with truss, I would see it do some stuff, fork, wait several seconds, then do some more stuff, rinse and repeat. Using 'truss -f' to follow the child processes, there was a noticeable delay associated with each fork() call. > > This led me to do some benchmarking. I found a fork() benchmark at [1] and ran it on various systems. Notably, on FreeBSD 10.1 (also 10.0) under Xen, it was reasonably fast shortly after bootup (though still slower than 9.3), but would get slower on repeated runs, and significantly slower after compiling some ports. It would also run slowly if the system had booted and then sat idle for a while. The speed was inconsistent, as occasionally after a period of idleness it would run somewhat faster again without rebooting; also configure and compilation times of sssd were inconsistent, but generally "slow", sometimes drastically so. > > FreeBSD 9.3 (with "xenhvm_load="YES" in loader.conf) on the same Xen host does not have this problem -- it fork()s more quickly and consistently; FreeBSD 10.1 on KVM (unfortunately not on the same hardware) also appears normal, as does 8.4 on (different but similar vintage) physical hardware, and a Linux VM on the same Xen host. Using one or two virtual CPUs does not make much difference, and the host machine is otherwise idle, so it does not appear to be an SMP issue. I was using ZFS, but I have ruled that out as a factor, as the problem occurs even without zfs.ko loaded (/ is ufs). Varying the memory between 1 and 8 GB did not seem to affect anything either. I also built a "NOHVM" 10.1 kernel to see if the Xen drivers were at issue, but that did not help (it was actually a bit slower), so it appears to be something deeper in the kernel or scheduler. > > The Xen host is running Xen 4.2.5_02-0.7.1 with SLES 11 SP3 as the Dom0, on a Dell 2950 with 8 physical CPU cores (dual socket, quad-core Xeon E5420). I have not experienced performance problems with any other guest OS. > > As FreeBSD 9.3 runs fine, I am using that for my FreeBSD VMs for now, but hopefully 10.x can be fixed before 9-STABLE goes EOL! Following are the VM config, dmesg, and some benchmarks. Thanks for the detailed description of the issue. I've reproduced your benchmarks and with the hardware I've used (Core i3-5010U) I'm not able to see this performance issue, below are the figures in my case: FreeBSD 9.3 after boot fork 1000 0.538132 FreeBSD 10.1 after boot fork 1000 0.514381 FreeBSD 9.3 build sssd 2098.66 real 1726.37 user 446.76 sys FreeBSD 10.1 build sssd 1247.38 real 1180.35 user 294.41 sys FreeBSD 9.3 after sssd fork 1000 0.543631 FreeBSD 10.1 after sssd fork 1000 0.538207 I'm Ccing feld because IIRC he found something similar on one of his boxes, that also had VTx but no EPT (just like yours). Would it be possible for you to try the same set of tests on a different hardware? Also, if even FreeBSD 10.1 compiled without XENHVM shows this issue it means there's something in the generic code that doesn't work well when running virtualized on this specific hardware, but I'm afraid figuring it out is not trivial. One place to start would be asking on freebsd-hackers and freebsd-virt. Roger.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5516A998.10206>