Date: Thu, 19 Sep 2013 11:25:40 -0500 From: Guy Helmer <guy.helmer@gmail.com> To: freebsd-hackers@freebsd.org Subject: IO Performance under VMware on LSI RAID controller Message-ID: <EC847303-6F76-44DE-916E-3C1C04F34773@gmail.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Normally I build VMware ESXi servers with enterprise-class WD SATA = drives and I/O performance in FreeBSD VMs on the servers is fine. Whenever I build a VMware ESXi server with a RAID controller, IO = performance is awful in FreeBSD VMs. I've previously seen this effect = with VMware ESXi 3ware 9690SA-8I and 9650 RAID controllers, and now I'm = seeing similar performance with a Dell 6/iR controller. Any suggestions would be appreciated. Guy Details of the current environment: VMware ESXi 5.1 on Dell R610 4GB = RAM, SAS 6/iR controller, 2x500GB disks in RAID1 set (default stripe = size) and 1x1TB (no RAID). =46rom VMware's client, I see I/O rates in = the sub-MBps range and latencies peaking occasionally at 80 ms. FreeBSD 9.2 (RC2) amd64 in a VM with 2GB RAM assigned, virtual disks = assigned from both the RAID1 set and 1TB (no RAID) drive, UFS+soft = updates file systems. The virtual drives show up in FreeBSD attached to an mpt virtual = controller: mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem = 0xd0040000-0xd005ffff,0xd0020000-0xd003ffff irq 17 at device 16.0 on = pci0 mpt0: MPI Version=3D1.2.0.0 I don't see anything else sharing the interrupt - vmstat -i shows: irq17: mpt0 77503 27 gstat is showing an abysmal 6 to 16 ops/s for requests on the virtual = disks. I've used gpart to setup the GPT partition table on the virtual disk = assigned from the 1TB drive with alignment for the first UFS partition = at 1MB to try to optimize alignment: Geom name: da0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 268435422 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da0p1 Mediasize: 524288 (512k) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r0w0e0 rawuuid: d9e6e3e8-1bdb-11e3-b7c5-000c29cbf143 rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f label: gpboot length: 524288 offset: 20480 type: freebsd-boot index: 1 end: 1063 start: 40 2. Name: da0p2 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 1048576 Mode: r1w1e2 rawuuid: fbd6cf40-1bdb-11e3-b7c5-000c29cbf143 rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: gprootfs length: 2147483648 offset: 1048576 type: freebsd-ufs index: 2 end: 4196351 start: 2048 3. Name: da0p3 Mediasize: 4294967296 (4.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2148532224 Mode: r1w1e1 rawuuid: 0658208d-1bdc-11e3-b7c5-000c29cbf143 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: gpswap length: 4294967296 offset: 2148532224 type: freebsd-swap index: 3 end: 12584959 start: 4196352 4. Name: da0p4 Mediasize: 130995437056 (122G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2148532224 Mode: r1w1e2 rawuuid: 0ca5bc32-1bdc-11e3-b7c5-000c29cbf143 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: gpusrfs length: 130995437056 offset: 6443499520 type: freebsd-swap index: 4 end: 268435422 start: 12584960 Consumers: 1. Name: da0 Mediasize: 137438953472 (128G) Sectorsize: 512 Mode: r3w3e8 sysctl vfs shows: vfs.ufs.dirhash_reclaimage: 5 vfs.ufs.dirhash_lowmemcount: 179 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 0 vfs.ufs.dirhash_maxmem: 3481600 vfs.ufs.dirhash_minsize: 2560 vfs.ufs.rename_restarts: 0 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.keytab_enctype: 1 vfs.nfs.skip_wcc_data_onerr: 1 vfs.nfs.nfs3_jukebox_delay: 10 vfs.nfs.reconnects: 0 vfs.nfs.bufpackets: 4 vfs.nfs.debuglevel: 0 vfs.nfs.callback_addr:=20 vfs.nfs.realign_count: 0 vfs.nfs.realign_test: 0 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfs.nfs_keep_dirty_on_error: 0 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.commit_on_close: 0 vfs.nfs.prime_access_cache: 0 vfs.nfs.access_cache_timeout: 60 vfs.nfs.diskless_rootpath:=20 vfs.nfs.diskless_valid: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.defect: 0 vfs.nfs.iodmax: 20 vfs.nfs.iodmin: 0 vfs.nfs.iodmaxidle: 120 vfs.devfs.rule_depth: 1 vfs.devfs.generation: 113 vfs.nfsd.disable_checkutf8: 0 vfs.nfsd.server_max_nfsvers: 4 vfs.nfsd.server_min_nfsvers: 2 vfs.nfsd.nfs_privport: 0 vfs.nfsd.async: 0 vfs.nfsd.enable_locallocks: 0 vfs.nfsd.issue_delegations: 0 vfs.nfsd.commit_miss: 0 vfs.nfsd.commit_blks: 0 vfs.nfsd.mirrormnt: 1 vfs.nfsd.minthreads: 1 vfs.nfsd.maxthreads: 1 vfs.nfsd.threads: 0 vfs.nfsd.request_space_used: 0 vfs.nfsd.request_space_used_highest: 0 vfs.nfsd.request_space_high: 13107200 vfs.nfsd.request_space_low: 8738133 vfs.nfsd.request_space_throttled: 0 vfs.nfsd.request_space_throttle_count: 0 vfs.nfsd.fha.enable: 1 vfs.nfsd.fha.bin_shift: 22 vfs.nfsd.fha.max_nfsds_per_fh: 8 vfs.nfsd.fha.max_reqs_per_nfsd: 0 vfs.nfsd.fha.fhe_stats: No file handle entries. vfs.pfs.trace: 0 vfs.pfs.vncache.misses: 0 vfs.pfs.vncache.hits: 0 vfs.pfs.vncache.maxentries: 0 vfs.pfs.vncache.entries: 0 vfs.acl_nfs4_old_semantics: 0 vfs.flushwithdeps: 0 vfs.unmapped_buf_allowed: 1 vfs.barrierwrites: 1 vfs.notbufdflashes: 0 vfs.flushbufqtarget: 100 vfs.mappingrestarts: 0 vfs.getnewbufrestarts: 337501 vfs.getnewbufcalls: 349444 vfs.hifreebuffers: 1524 vfs.lofreebuffers: 762 vfs.numfreebuffers: 13601 vfs.dirtybufthresh: 3084 vfs.hidirtybuffers: 3427 vfs.lodirtybuffers: 1713 vfs.numdirtybuffers: 11 vfs.recursiveflushes: 341175 vfs.altbufferflushes: 0 vfs.bdwriteskip: 0 vfs.dirtybufferflushes: 0 vfs.hirunningspace: 3538944 vfs.lorunningspace: 2359296 vfs.bufdefragcnt: 0 vfs.buffreekvacnt: 339965 vfs.bufreusecnt: 347292 vfs.hibufspace: 222625792 vfs.lobufspace: 222560256 vfs.maxmallocbufspace: 11131289 vfs.bufmallocspace: 0 vfs.maxbufspace: 223281152 vfs.unmapped_bufspace: 290652160 vfs.bufspace: 291602432 vfs.runningbufspace: 131072 vfs.vmiodirenable: 1 vfs.cache.numfullpathfound: 47 vfs.cache.numfullpathfail4: 0 vfs.cache.numfullpathfail2: 0 vfs.cache.numfullpathfail1: 0 vfs.cache.numfullpathcalls: 47 vfs.cache.numupgrades: 32 vfs.cache.numneghits: 2853 vfs.cache.numnegzaps: 16 vfs.cache.numposhits: 356750 vfs.cache.numposzaps: 282 vfs.cache.nummisszap: 10 vfs.cache.nummiss: 35615 vfs.cache.numchecks: 377860 vfs.cache.dotdothits: 19 vfs.cache.dothits: 146 vfs.cache.numcalls: 395710 vfs.cache.numcache: 29909 vfs.cache.numneg: 336 vfs.ncsizefactor: 2 vfs.ncnegfactor: 16 vfs.read_min: 1 vfs.read_max: 64 vfs.write_behind: 1 vfs.typenumhash: 1 vfs.lookup_shared: 1 vfs.usermount: 0 vfs.worklist_len: 3 vfs.timestamp_precision: 0 vfs.reassignbufcalls: 515051 vfs.vlru_allow_cache_src: 0 vfs.freevnodes: 27829 vfs.wantfreevnodes: 27833 vfs.numvnodes: 29507 vfs.ffs.doreallocblks: 1 vfs.ffs.doasyncfree: 1 vfs.ffs.compute_summary_at_mount: 0 --Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJSOyWEAAoJEJmJzK+PHOvEhgsH/0qm0ZD7qZ0CHhIVxSBn9f4Y eY5HpTRbjXWEtG0En3V8ZcLjk/oXF6CSC6olwvczim6pscq3XJ2EDK+6zWZFHYq8 KczXW++3T0U5HxK83ghjFZidsgo64lKFKpQb35KOZFd0MIWHHYYRx9EL+uFdyxXn BycgjQral+gZqwRELYwOtA5oqGXkMQpAIh18X1qH/ZVPD3XFwewsDe0VZuf4jHqi rZFMozmvDyqfP2ytYFWNHuO3U1OHbn6yRrwfCuLIHmD498ms/FYTwtEVTZ7xQr0n lxofZVQcI+t9XLe36n7Jgci6+YVL1N2lI2KZ5/t8QPwLV5Po1N2/nbGTI+3Rjac= =tq1b -----END PGP SIGNATURE----- --Apple-Mail=_AA5E5FCA-97F8-4583-9505-7A8B5C122F2E--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EC847303-6F76-44DE-916E-3C1C04F34773>