Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 11 Oct 2006 14:24:29 -0400
From:      Jeff Dickens <jeff@intranet.seamanpaper.com>
Cc:        freebsd-questions@freebsd.org
Subject:   Re: optimal kernel options for VMWARE guest system
Message-ID:  <452D36DD.1070800@intranet.seamanpaper.com>
In-Reply-To: <4523C9C2.6060000@seamanpaper.com>
References:  <4522969F.9010504@seamanpaper.com>	<200610031605.54121.lists@jnielsen.net> <4523C9C2.6060000@seamanpaper.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
Jeff Dickens wrote:
> John Nielsen wrote:
>> On Tuesday 03 October 2006 12:58, Jeff Dickens wrote:
>>  
>>> I have some Freebsd systems that are running as VMware guests.  I'd 
>>> like
>>> to configure their kernels so as to minimize the overhead on the VMware
>>> host system.  After reading and partially digesting the white paper on
>>> timekeeping in VMware virtual machines
>>> (http://www.vmware.com/pdf/vmware_timekeeping.pdf) it appears that I
>>> might want to make some changes.
>>>
>>> Has anyone addressed this issue?
>>>     
>>
>> I haven't read the white paper (yet; thanks for the link), but I've 
>> had good results with recent -STABLE VM's running under ESX server 3. 
>> Some thoughts:
>>
>> As I do on most of my installs, I trimmed down GENERIC to include 
>> just the drivers I use. In this case that was mpt for the disk and le 
>> for the network (although I suspect forcing the VM to present e1000 
>> hardware and then using the em driver would work as well if not better).
>>
>> The VMware tools package that comes with ESX server does a poor job 
>> of getting itself to run, but it can be made to work without too much 
>> difficulty. Don't use the port, run the included install script to 
>> install the files, ignore the custom network driver and compile the 
>> memory management module from source (included). If using X.org, use 
>> the built-in vmware display driver, and copy the vmmouse driver .o 
>> file from the VMware tools dist to the appropriate dir under 
>> /usr/X11. Even though the included file is for X.org 6.8, it works 
>> fine with 6.9/7.0 (X.org 7.1 should include the vmmouse driver.) Run 
>> the VMware tools config script from a non-X terminal (and you can 
>> ignore the warning about running it remotely if you're using SSH), so 
>> it won't mess with your X display (it doesn't do anything not 
>> accomplished above). Then run the rc.d script to start the VMware tools.
>>
>> I haven't noticed any timekeeping issues so far.
>>
>> JN
>> _______________________________________________
>>   
> What is the advantage of using the "e1000 hardware", and is this 
> documented somewhere?  I got the vxn network driver working without 
> issues; I just had to edit the .vxn file manually:  I'm using the free 
> VMware server V1 rather than the ESX server.
>
>   ethernet0.virtualDev="vmxnet"
>
> I've got timekeeping running stably on these.  I turn on time sync via 
> vmware tools in the .vmx file:
>
>  tools.syncTime = "TRUE"
>
> and in the guest file's rc.conf start ntpd with flags "-Aqgx &" so it 
> just syncs once at boot and exits.
>
> I'm not using X on these.  They're supposed to be clean & lean systems 
> to run such things as djbdns and qmail.  And they do work well. 
> My main goal is to reduce the background load on the VMware host 
> system so that it isn't spending more time than it has to simulating 
> interrupt controllers for the guests.  I'm wondering about the 
> "disable ACPI" boot option.  I suppose I first should figure out how 
> to even roughly measure the effect of any changes I might make.
>
Well, I've done some pseudo-scientific measurement on this.  I currently 
have five freebsd virtual systems running, and one Centos 4 (linux 
2.6),   This command give some info on the background cpu usage:

(The host is a Centos 3 system, linux 2.4)

[root@otter root]# ps auxww | head -1
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
[root@otter root]# ps auxww | grep vmx
root     18031 12.7  1.5 175440 39916 ?      S<   Oct09 345:50 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/Goose/freebsd-6.1-i386.vmx -@ ""
root     18058 12.9  1.4 174772 36916 ?      S<   Oct09 351:01 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/Duck/freebsd-6.1-i386.vmx -@ ""
root     18072 16.2  5.5 246372 141776 ?     S<   Oct09 440:16 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/BlueJay/freebsd-6.1-i386.vmx -@ ""
root     18086 12.9  1.4 174688 38464 ?      S<   Oct09 351:47 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/Heron/freebsd-6.1-i386.vmx -@ ""
root     18100  9.4  4.1 385712 107348 ?     S<   Oct09 256:25 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/Newt/freebsd-6.1-i386.vmx -@ ""
root     18139 12.2  2.5 299388 65132 ?      S<   Oct09 330:35 
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual 
Machines/Centos4/Centos4.vmx -@ ""
root     28930  0.0  0.0  3680  672 pts/3    S    14:08   0:00 grep vmx
[root@otter root]#


As one can see the one called "Newt" is consistently lower in the "%CPU" 
column.  Curiously enough, this *is* the one I built a custom kernel for. 

The config file I used is posted below:  Besides commenting out devices 
I wasn't using & NFS, etc, I commented out the apic and pctimer 
devices.  Do you think I'm on the right track for reducing interrupt 
frequency?

Also, if I were to want to move this kernel to other FreeBSD systems, 
how much has to move, the whole /boot/kernel directory?

Finally I did have to re-run the vmware-config-tools.pl script after 
rebuilding the kernel.


newt# cat VMWARE1
#
# GENERIC -- Generic kernel configuration file for FreeBSD/i386
#
# For more information on this file, please read the handbook section on
# Kernel Configuration Files:
#
#    
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.429.2.7.2.2 2006/05/01 
00:15:12 scottl Exp $

machine         i386
cpu             I686_CPU
ident           VMWARE1

# To statically compile in device wiring instead of /boot/device.hints
#hints          "GENERIC.hints"         # Default places to look for 
devices.

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug 
symbols

#options        SCHED_ULE               # ULE scheduler
options         SCHED_4BSD              # 4BSD scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
#options        INET6                   # IPv6 communications protocols
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big 
directories
options         MD_ROOT                 # MD is a potential root device
#options        NFSCLIENT               # Network Filesystem Client
#options        NFSSERVER               # Network Filesystem Server
#options        NFS_ROOT                # NFS usable as /, requires 
NFSCLIENT
#options        MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires 
PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_GPT                # GUID Partition Tables.
options         COMPAT_43               # Compatible with BSD 4.3 [KEEP 
THIS!]
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time 
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         AHC_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~128k to driver.
options         AHD_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~215k to driver.
options         ADAPTIVE_GIANT          # Giant mutex is adaptive.

#device         apic                    # I/O APIC

# Bus support.
#device         eisa
device          pci

# Floppy drives
#device         fdc

# ATA and ATAPI devices
device          ata
device          atadisk         # ATA disk drives
#device         ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
#device         atapifd         # ATAPI floppy drives
#device         atapist         # ATAPI tape drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI Controllers
#device         ahb             # EISA AHA1742 family
#device         ahc             # AHA2940 and onboard AIC7xxx devices
#device         ahd             # AHA39320/29320 and onboard AIC79xx devices
#device         amd             # AMD 53C974 (Tekram DC-390(T))
#device         isp             # Qlogic family
##device        ispfw           # Firmware for QLogic HBAs- normally a 
module
#device         mpt             # LSI-Logic MPT-Fusion
##device                ncr             # NCR/Symbios Logic
#device         sym             # NCR/Symbios Logic (newer chipsets + 
those of `ncr')
#device         trm             # Tekram DC395U/UW/F DC315U adapters

#device         adv             # Advansys SCSI adapters
#device         adw             # Advansys wide SCSI adapters
#device         aha             # Adaptec 154x SCSI adapters
#device         aic             # Adaptec 15[012]x SCSI adapters, 
AIC-6[23]60.
#device         bt              # Buslogic/Mylex MultiMaster SCSI adapters

#device         ncv             # NCR 53C500
#device         nsp             # Workbit Ninja SCSI-3
#device         stg             # TMC 18C30/18C50

# SCSI peripherals
#device         scbus           # SCSI bus (required for SCSI)
#device         ch              # SCSI media changers
#device         da              # Direct Access (disks)
#device         sa              # Sequential Access (tape etc)
#device         cd              # CD
#device         pass            # Passthrough device (direct SCSI access)
#device         ses             # SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
#device         amr             # AMI MegaRAID
#device         arcmsr          # Areca SATA II RAID
#device         asr             # DPT SmartRAID V, VI and Adaptec SCSI RAID
#device         ciss            # Compaq Smart RAID 5*
#device         dpt             # DPT Smartcache III, IV - See NOTES for 
options
#device         hptmv           # Highpoint RocketRAID 182x
#device         rr232x          # Highpoint RocketRAID 232x
#device         iir             # Intel Integrated RAID
#device         ips             # IBM (Adaptec) ServeRAID
#device         mly             # Mylex AcceleRAID/eXtremeRAID
#device         twa             # 3ware 9000 series PATA/SATA RAID

# RAID controllers
#device         aac             # Adaptec FSA RAID
#device         aacp            # SCSI passthrough for aac (requires CAM)
#device         ida             # Compaq Smart RAID
#device         mfi             # LSI MegaRAID SAS
#device         mlx             # Mylex DAC960 family
#device         pst             # Promise Supertrak SX6000
#device         twe             # 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          kbdmux          # keyboard multiplexer

device          vga             # VGA video card driver

device          splash          # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc

# Enable this for the pcvt (VT220 compatible) console driver
#device         vt
#options        XSERVER         # support for X server on a vt console
#options        FAT_CURSOR      # start with block cursor

#device         agp             # support several AGP chipsets

# Power management support (see NOTES for more options)
#device         apm
# Add suspend/resume support for the i8254.
#device         pmtimer

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device         cbb             # cardbus (yenta) bridge
#device         pccard          # PC Card (16-bit) bus
#device         cardbus         # CardBus (32-bit) bus

# Serial (COM) ports
#device         sio             # 8250, 16[45]50 based serial ports

# Parallel port
#device         ppc
#device         ppbus           # Parallel port bus (required)
#device         lpt             # Printer
#device         plip            # TCP/IP over parallel
#device         ppi             # Parallel port interface device
##device                vpo             # Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device         puc

# PCI Ethernet NICs.
#device         de              # DEC/Intel DC21x4x (``Tulip'')
#device         em              # Intel PRO/1000 adapter Gigabit 
Ethernet Card
#device         ixgb            # Intel PRO/10GbE Ethernet Card
#device         txp             # 3Com 3cR990 (``Typhoon'')
device          vx              # 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support
#device         bce             # Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device         bfe             # Broadcom BCM440x 10/100 Ethernet
#device         bge             # Broadcom BCM570xx Gigabit Ethernet
#device         dc              # DEC/Intel 21143 and various workalikes
#device         fxp             # Intel EtherExpress PRO/100B (82557, 82558)
#device         lge             # Level 1 LXT1001 gigabit Ethernet
#device         nge             # NatSemi DP83820 gigabit Ethernet
#device         nve             # nVidia nForce MCP on-board Ethernet 
Networking
#device         pcn             # AMD Am79C97x PCI 10/100(precedence 
over 'lnc')
#device         re              # RealTek 8139C+/8169/8169S/8110S
#device         rl              # RealTek 8129/8139
#device         sf              # Adaptec AIC-6915 (``Starfire'')
#device         sis             # Silicon Integrated Systems SiS 900/SiS 
7016
#device         sk              # SysKonnect SK-984x & SK-982x gigabit 
Ethernet
#device         ste             # Sundance ST201 (D-Link DFE-550TX)
#device         ti              # Alteon Networks Tigon I/II gigabit 
Ethernet
#device         tl              # Texas Instruments ThunderLAN
#device         tx              # SMC EtherPower II (83c170 ``EPIC'')
#device         vge             # VIA VT612x gigabit Ethernet
#device         vr              # VIA Rhine, Rhine II
#device         wb              # Winbond W89C840F
#device         xl              # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.  pccard NICs included.
#device         cs              # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device         ed              # NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device         ex              # Intel EtherExpress Pro/10 and Pro/10+
#device         ep              # Etherlink III based cards
#device         fe              # Fujitsu MB8696x based cards
#device         ie              # EtherExpress 8/16, 3C507, StarLAN 10 etc.
device          lnc             # NE2100, NE32-VL Lance Ethernet cards
#device         sn              # SMC's 9000 series of Ethernet chips
#device         xe              # Xircom pccard Ethernet

# Wireless NIC cards
#device         wlan            # 802.11 support
#device         an              # Aironet 4500/4800 802.11 wireless NICs.
#device         awi             # BayStack 660 and others
#device         ral             # Ralink Technology RT2500 wireless NICs.
#device         wi              # WaveLAN/Intersil/Symbol 802.11 
wireless NICs.
##device                wl              # Older non 802.11 Wavelan 
wireless NIC.

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
#device         sl              # Kernel SLIP
#device         ppp             # Kernel PPP
#device         tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
#device         gif             # IPv6 and IPv4 tunneling
#device         faith           # IPv6-to-IPv4 relaying (translation)

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter

# USB support
#device         uhci            # UHCI PCI->USB interface
#device         ohci            # OHCI PCI->USB interface
#device         ehci            # EHCI PCI->USB interface (USB 2.0)
#device         usb             # USB Bus (required)
##device                udbp            # USB Double Bulk Pipe devices
#device         ugen            # Generic
#device         uhid            # "Human Interface Devices"
#device         ukbd            # Keyboard
#device         ulpt            # Printer
#device         umass           # Disks/Mass storage - Requires scbus and da
#device         ums             # Mouse
#device         ural            # Ralink Technology RT2500USB wireless NICs
#device         urio            # Diamond Rio 500 MP3 player
#device         uscanner        # Scanners
# USB Ethernet, requires miibus
#device         aue             # ADMtek USB Ethernet
#device         axe             # ASIX Electronics USB Ethernet
#device         cdce            # Generic USB over Ethernet
#device         cue             # CATC USB Ethernet
#device         kue             # Kawasaki LSI USB Ethernet
#device         rue             # RealTek RTL8150 USB Ethernet

# FireWire support
#device         firewire        # FireWire bus code
#device         sbp             # SCSI over FireWire (Requires scbus and da)
#device         fwe             # Ethernet over FireWire (non-standard!)
newt#





Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?452D36DD.1070800>