Date: Tue, 3 May 2011 14:34:39 -0400 (GMT-04:00) From: John <aqqa11@earthlink.net> To: freebsd-emulation@freebsd.org Subject: Re: virtualbox I/O 3 times slower than KVM? Message-ID: <20037078.1304447680252.JavaMail.root@mswamui-blood.atl.sa.earthlink.net>
next in thread | raw e-mail | index | archive | help
-----Original Message----- >From: John <aqqa11@earthlink.net> >Sent: May 3, 2011 2:25 PM >To: >Cc: freebsd-emulation@freebsd.org >Subject: Re: virtualbox I/O 3 times slower than KVM? > > >-----Original Message----- >>From: Ted Mittelstaedt <tedm@mittelstaedt.us> >>Sent: May 3, 2011 12:02 AM >>To: Adam Vande More <amvandemore@gmail.com> >>Cc: freebsd-emulation@freebsd.org >>Subject: Re: virtualbox I/O 3 times slower than KVM? >> >>On 5/2/2011 7:39 PM, Adam Vande More wrote: >>> On Mon, May 2, 2011 at 4:30 PM, Ted Mittelstaedt <tedm@mittelstaedt.us >>> <mailto:tedm@mittelstaedt.us>> wrote: >>> >>> that's sync within the VM. Where is the bottleneck taking place? If >>> the bottleneck is hypervisor to host, then the guest to vm write may >>> write all it's data to a memory buffer in the hypervisor that is >>> then slower-writing it to the filesystem. In that case killing the >>> guest without killing the VM manager will allow the buffer to >>> complete emptying since the hypervisor isn't actually being shut down. >>> >>> >>> No the bottle neck is the emulated hardware inside the VM process >>> container. This is easy to observe, just start a bound process in the >>> VM and watch top host side. Also the hypervisor uses native host IO >>> driver, there's no reason for it to be slow. Since it's the emulated >>> NIC which is the bottleneck, there is nothing left to issue the write. >>> Further empirical evidence for this can be seen by by watching gstat on >>> VM running with an md or ZVOL backed storage. I already utilize ZVOL's >>> for this so it was pretty easy to confirm no IO occurs when the VM is >>> paused or shutdown. >>> >>> Is his app going to ever face the extremely bad scenario, though? >>> >>> >>> The point is it should be relatively easy to induce patterns you expect >>> to see in production. If you can't, I would consider that a problem. >>> Testing out theories(performance based or otherwise) on a production >>> system is not a good way to keep the continued faith of your clients >>> when the production system is a mission critical one. Maybe throwing >>> more hardware at a problem is the first line of defense for some >>> companies, unfortunately I don't work for them. Are they hiring? ;) I >>> understand the logic of such an approach and have even argued for it >>> occasionally. Unfortunately payroll is already in the budget, extra >>> hardware is not even if it would be a net savings. >>> >> >>Most if not all sites I've ever been in that run Windows servers behave >>in this manner. With most of these sites SOP is to "prove" that the >>existing hardware is inadequate by loading whatever Windows software >>that management wants loaded then letting the users on the network >>scream about it. Then money magically frees itself up when there wasn't >>any before. Since of course management will never blame the OS for >>the slowness, always the hardware. >> >>Understand I'm not advocating this, just making an observation. >> >>Understand that I'm not against testing but I've seen people get >>so engrossed in spending time constructing test suites that they >>have ended up wasting a lot of money. I would have to ask, how much >>time did the OP who started this thread take building 2 systems, >>a Linux and a BSD system? How much time has he spent trying to get >>the BSD system to "work as well as the Linux" system? Wouldn't it >>have been cheaper for him to not spend that time and just put the >>Linux system into production? >> >>Ted > >Thanks a lot for everyone's insights and suggestions. The CentOS on the KVM is a production server, so I took some time to prepare another CentOS on that KVM and did the test as Ted suggested before (for comparison, right now the test freebsd is the only guest on the virtualbox). > >What I do is to cat the 330MB binary file (XP service pack from Microsoft) 20 times into a single 6.6GB file, "date" before and afterwards, and after the second date finishes, immediately Force power shut down. There are two observations: > >1. the time to complete copying into this 6.6GB file were 72s, 44s, 79s in three runs, presumably because there is another production VM on the same host. The average is 65s, so it's about 100MB/s. >2. After immediately power down, I do found the resulting file was less than 6.6GB. So indeed the VM claimed the completion of the copying before it actually did. > >I then did the same thing on the virtualbox, since I don't want the above premature I/O, I made sure the "Use Host I/O cache" is unchecked for the VM storage. > >1. the time to complete copying into this 6.6GB file was 119s and 92s, the average is 105s, so the speed is 62MB/s. >2. after immediately "Reset" the machine, I couldn't boot. Both times it asked me to do fsck for that partition (GPT 2.2T). But after finally powering up, I found the file was also less than 6.6GB both times as well. > >So looks like virtualbox also suffers caching problem? Or did I do anything wrong? > >I didn't spend extra time optimizing either the linux or the freebsd, they are both the production systems from centos and freebsd. I just want to have a production quality system without too much customized work. > >Also, most servers will be mail servers and web servers, with some utilization of database. Granted, copying 6.6GB file is atypical on these servers, but I just want to get an idea of what the server is capable of. I do not know a test software that can benchmark my usage pattern and is readily available on both centos and freebsd. By the way, I do use bridged networking (it's so much easier to configure bridged network on virtualbox than on kvm). Network on the VM is fine to me. Right now the issue is the I/O.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20037078.1304447680252.JavaMail.root>