Date: Thu, 20 Nov 2014 19:27:48 +0100 From: Ewald Jenisch <a@jenisch.at> To: <freebsd-questions@freebsd.org> Subject: SCSI errors on VMware guest Message-ID: <20141120182748.GA3546@aurora.oekb.co.at>
next in thread | raw e-mail | index | archive | help
Hi, I'm running a FreeBSD 9.3 machine (amd64) on a VMware 5.5 host. "Disks" used by this VM come from a NetApp system providing storage for 200+ virtual machines. Increasingly I'm seeing errors like these: Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): WRITE(10). CDB: 2a 00 00 20 02 a2 00 00 40 00 Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): CAM status: SCSI Status Error Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): SCSI status: Busy Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): Retrying command Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): WRITE(10). CDB: 2a 00 00 c0 00 a2 00 00 08 00 Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): CAM status: SCSI Status Error Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): SCSI status: Busy Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): CAM status: SCSI Status Error Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): SCSI status: Busy Nov 19 09:02:59 igue kernel: (da0:mpt0:0:0:0): Retrying command Eventually this ends up in a crash of the system: Nov 19 09:48:17 igue syslogd: kernel boot file is /boot/kernel/kernel Nov 19 09:48:17 igue kernel: panic: initiate_write_inodeblock_ufs2: already started Nov 19 09:48:17 igue kernel: cpuid = 0 Nov 19 09:48:17 igue kernel: KDB: stack backtrace: Nov 19 09:48:17 igue kernel: #0 0xffffffff80934766 at kdb_backtrace+0x66 Nov 19 09:48:17 igue kernel: #1 0xffffffff808fa2ee at panic+0x1ce Nov 19 09:48:17 igue kernel: #2 0xffffffff80b35814 at softdep_disk_io_initiation+0xf04 Nov 19 09:48:17 igue kernel: #3 0xffffffff80b4352f at ffs_geom_strategy+0x17f Nov 19 09:48:17 igue kernel: #4 0xffffffff80980195 at bufwrite+0x145 Nov 19 09:48:17 igue kernel: #5 0xffffffff80979fdf at vfs_bio_awrite+0x7f Nov 19 09:48:17 igue kernel: #6 0xffffffff80986448 at vop_stdfsync+0x288 Nov 19 09:48:17 igue kernel: #7 0xffffffff807dc5e8 at devfs_fsync+0x98 Nov 19 09:48:17 igue kernel: #8 0xffffffff80de31c8 at VOP_FSYNC_APV+0x78 Nov 19 09:48:17 igue kernel: #9 0xffffffff8099ab5b at sync_vnode+0x16b Nov 19 09:48:17 igue kernel: #10 0xffffffff8099ae65 at sched_sync+0x1c5 Nov 19 09:48:17 igue kernel: #11 0xffffffff808c810f at fork_exit+0x11f Nov 19 09:48:17 igue kernel: #12 0xffffffff80ccc5be at fork_trampoline+0xe Nov 19 09:48:17 igue kernel: Uptime: 12d20h44m1s Nov 19 09:48:17 igue kernel: (da0:mpt0:0:0:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 Nov 19 09:48:17 igue kernel: (da0:mpt0:0:0:0): CAM status: Command timeout Nov 19 09:48:17 igue kernel: (da0:mpt0:0:0:0): Error 5, Retries exhausted Nov 19 09:48:17 igue kernel: (da0:mpt0:0:0:0): Synchronize cache failed I've fsck-ed all filesystems in singleuser to be sure there's no logical error, but these SCSI error messages keep coming - and with them crashes of the VM. Also "vmware-checkvm" means everything's fine # vmware-checkvm -h VM's hw version is 4 VMware software version 6 (good) # BTW, I've installed vmware tools that come with the vmware host following this http://hephaex.blogspot.co.at/2013/01/installing-vmware-tools-on-freebsd-91.html advice, i.e. pkg_add -r perl cd /usr/ports/misc/compat6x/; make && make install mount -t cd9660 /dev/cd0 /cdrom cd /tmp mkdir VMware-tools cd VMware-tools/ cp /cdrom/vmware-freebsd-tools.tar.gz . tar zxf vmware-freebsd-tools.tar.gz cd vmware-tools-distrib/lib/modules/source/ tar xf vmmemctl.tar cd vmmemctl-only/ make; make install cd /tmp/VMware-tools/vmware-tools-distrib/lib/modules/source/ tar xf vmblock.tar cd vmblock-only make && make install cd /tmp/VMware-tools/vmware-tools-distrib/ ./vmware-install.pl No errors during building and/or installing. BTW, I've seen these errors only on a FreeBSD system - no other VM running on the host has reported any error whatsoever; so to my understanding it's a FreeBSD related thing going on here.2 Has anybody else seen these errors? Any known cure against it? Thanks much in advance for any clue, -ewald
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141120182748.GA3546>