From owner-freebsd-xen@FreeBSD.ORG Thu Oct 9 14:02:25 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62350559 for ; Thu, 9 Oct 2014 14:02:25 +0000 (UTC) Received: from ezwind.net (bobby.ezwind.net [199.188.211.146]) by mx1.freebsd.org (Postfix) with ESMTP id 3593B18A for ; Thu, 9 Oct 2014 14:02:24 +0000 (UTC) Received: from jayPC by ezwind.net (MDaemon PRO v9.6.5) with ESMTP id 50-md50000166449.msg for ; Thu, 09 Oct 2014 09:01:58 -0500 X-Spam-Processed: ezwind.net, Thu, 09 Oct 2014 09:01:58 -0500 (not processed: spam filter heuristic analysis disabled) X-Authenticated-Sender: jwest@ezwind.net X-MDRemoteIP: 97.91.122.42 X-Return-Path: prvs=1359fe52bd=jwest@ezwind.net X-Envelope-From: jwest@ezwind.net X-MDaemon-Deliver-To: freebsd-xen@freebsd.org From: "Jay West" To: Subject: disk loss Date: Thu, 9 Oct 2014 09:08:49 -0500 Message-ID: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac/jyUJg0cK7tcA7QO6WSikv6NFC7A== Content-Language: en-us Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2014 14:02:25 -0000 We're running lots of FreeBSD10 VM's under Xenserver 6.2, and using FreeNas 9.X for iscsi storage. Note, none of the VM's see iSCSI - just the hypervisors see it to create their storage repositories. We've noticed an ongoing issue with recovery from crashes or issues with the NAS. This doesn't happen often at all, but the nas will reboot about once a year. Windows VM's seem to recover fine. FreeBSD, not so much. The pattern seems to be that if the NAS holding the SR's for the hypervisor reboots, when the freebsd vm's come up that have just one virtual disk (boot), it complains about unclean shutdown and fsck needs to be run. No problem, running fsck fixes it. However, if the freebsd vm has two virtual disks (boot and data), the first one (say, ada0) complains about unclean shutdown and fsck will fix it. But the second disk (say, ada1) is not found. All we get is "can't stat /dev/ada1p1" or somesuch, and /dev/ad1 is missing from /dev. This seems to be a pretty consistent failure mode. We've tried to detach the data disk and attach it to a different VM - but no joy. 1) Anyone have any idea why FreeBSD seems so affected and Windows doesn't? 2) Any ideas as to how to (non-destructively) get /dev/ada1 back so we can try to access the data? Thanks in advance for any thoughts. J From owner-freebsd-xen@FreeBSD.ORG Thu Oct 9 14:26:58 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9ED3AA7 for ; Thu, 9 Oct 2014 14:26:58 +0000 (UTC) Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BBB505EA for ; Thu, 9 Oct 2014 14:26:58 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by gateway2.nyi.internal (Postfix) with ESMTP id D7AE7217B for ; Thu, 9 Oct 2014 10:26:56 -0400 (EDT) Received: from web6 ([10.202.2.216]) by compute2.internal (MEProxy); Thu, 09 Oct 2014 10:26:56 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:x-sasl-enc:from:to :mime-version:content-transfer-encoding:content-type:subject :date:in-reply-to:references; s=smtpout; bh=BmnbRoVYDfIYitUqOOpq NVAAoM0=; b=OjHP0sBYFzBGZLerm9d0ImI2nw77IjSeEPsPjxRyFxGySPHGhvwc xm0RNi6hToJ0gh60Dd0giw3LnfAFgoILo86lAoG2skaEShCgXv48CrhaOMVo59kv G/m4qavrwcPmY5J3ALCzOFPAyKXl9yS9a9hZnaTAn1y2jKviJT7so8A= Received: by web6.nyi.internal (Postfix, from userid 99) id 8842A962611; Thu, 9 Oct 2014 10:26:56 -0400 (EDT) Message-Id: <1412864816.1914399.177057565.4EBD1E5E@webmail.messagingengine.com> X-Sasl-Enc: Y2+l3MsW5g2KILg24w/+3s7Hr2ZMqHA7kKiITJOtjymJ 1412864816 From: Mark Felder To: freebsd-xen@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-1d265f39 Subject: Re: disk loss Date: Thu, 09 Oct 2014 09:26:56 -0500 In-Reply-To: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> References: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2014 14:26:59 -0000 On Thu, Oct 9, 2014, at 09:08, Jay West wrote: > We're running lots of FreeBSD10 VM's under Xenserver 6.2, and using > FreeNas > 9.X for iscsi storage. Note, none of the VM's see iSCSI - just the > hypervisors see it to create their storage repositories. > > > > We've noticed an ongoing issue with recovery from crashes or issues with > the > NAS. This doesn't happen often at all, but the nas will reboot about once > a > year. > > > > Windows VM's seem to recover fine. FreeBSD, not so much. The pattern > seems > to be that if the NAS holding the SR's for the hypervisor reboots, when > the > freebsd vm's come up that have just one virtual disk (boot), it complains > about unclean shutdown and fsck needs to be run. No problem, running fsck > fixes it. However, if the freebsd vm has two virtual disks (boot and > data), > the first one (say, ada0) complains about unclean shutdown and fsck will > fix > it. But the second disk (say, ada1) is not found. All we get is "can't > stat > /dev/ada1p1" or somesuch, and /dev/ad1 is missing from /dev. > > > > This seems to be a pretty consistent failure mode. We've tried to detach > the > data disk and attach it to a different VM - but no joy. > > > > 1) Anyone have any idea why FreeBSD seems so affected and Windows > doesn't? > > 2) Any ideas as to how to (non-destructively) get /dev/ada1 back so > we > can try to access the data? > > Are you rebooting the iSCSI NAS while the servers are running? Does rescanning the repository help at all? (or whatever the right-click option is on the SR... I don't have access to XenServer anymore and can't remember what that option is called) From owner-freebsd-xen@FreeBSD.ORG Thu Oct 9 15:41:37 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50C9C1EF; Thu, 9 Oct 2014 15:41:37 +0000 (UTC) Received: from ezwind.net (bobby.ezwind.net [199.188.211.146]) by mx1.freebsd.org (Postfix) with ESMTP id 19F17FEC; Thu, 9 Oct 2014 15:41:36 +0000 (UTC) Received: from jayPC by ezwind.net (MDaemon PRO v9.6.5) with ESMTP id 10-md50000166591.msg; Thu, 09 Oct 2014 10:41:35 -0500 X-Spam-Processed: ezwind.net, Thu, 09 Oct 2014 10:41:35 -0500 (not processed: spam filter heuristic analysis disabled) X-Authenticated-Sender: jwest@ezwind.net X-MDRemoteIP: 97.91.122.42 X-Return-Path: prvs=1359fe52bd=jwest@ezwind.net X-Envelope-From: jwest@ezwind.net From: "Jay West" To: "'Mark Felder'" , References: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> <1412864816.1914399.177057565.4EBD1E5E@webmail.messagingengine.com> In-Reply-To: <1412864816.1914399.177057565.4EBD1E5E@webmail.messagingengine.com> Subject: RE: disk loss Date: Thu, 9 Oct 2014 10:48:30 -0500 Message-ID: <000601cfe3d8$7a07b400$6e171c00$@ezwind.net> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQLZQaRIeWHWrqmNu5N4SPLKldtZ2QJB1s1umgMWvfA= Content-Language: en-us X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2014 15:41:37 -0000 Mark wrote... >Are you rebooting the iSCSI NAS while the servers are running? Not intentionally ;) On very rare occasion, an iscsi network cable gets tripped over, or for whatever reason the freenas box reboots itself. >Does rescanning the repository help at all? (or whatever the right-click option is on the SR... I don't have access to XenServer anymore and can't remember what that option is called) No luck :\ The VDI's are there, they just can be read by the guest OS. We've also tried detaching the vdi and attaching it to a different VM in an effort to read it, but no luck. _______________________________________________ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" From owner-freebsd-xen@FreeBSD.ORG Thu Oct 9 18:09:41 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6A94FC2 for ; Thu, 9 Oct 2014 18:09:41 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Cybertrust Public SureServer SV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 113C0382 for ; Thu, 9 Oct 2014 18:09:40 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.04,686,1406592000"; d="scan'208";a="181060934" Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.3.181.6; Thu, 9 Oct 2014 14:08:20 -0400 Message-ID: <5436CF13.4080509@citrix.com> Date: Thu, 9 Oct 2014 20:08:19 +0200 From: =?windows-1252?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Jay West , Subject: Re: disk loss References: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> In-Reply-To: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2014 18:09:41 -0000 El 09/10/14 a les 16.08, Jay West ha escrit: > We're running lots of FreeBSD10 VM's under Xenserver 6.2, and using FreeNas > 9.X for iscsi storage. Note, none of the VM's see iSCSI - just the > hypervisors see it to create their storage repositories. > > > > We've noticed an ongoing issue with recovery from crashes or issues with the > NAS. This doesn't happen often at all, but the nas will reboot about once a > year. > > > > Windows VM's seem to recover fine. FreeBSD, not so much. The pattern seems > to be that if the NAS holding the SR's for the hypervisor reboots, when the > freebsd vm's come up that have just one virtual disk (boot), it complains > about unclean shutdown and fsck needs to be run. No problem, running fsck > fixes it. However, if the freebsd vm has two virtual disks (boot and data), > the first one (say, ada0) complains about unclean shutdown and fsck will fix > it. But the second disk (say, ada1) is not found. All we get is "can't stat > /dev/ada1p1" or somesuch, and /dev/ad1 is missing from /dev. > > > > This seems to be a pretty consistent failure mode. We've tried to detach the > data disk and attach it to a different VM - but no joy. When this happens, can you log into the affected VM (with the disk "supposedly" attached) and paste the output of `xenstore-ls -fp /local/domain/` (replace with the ID of the domain) here. You will need the xen-tools package installed in order to do that. It looks to me like this is some kind of issue with the XenServer storage manager, so I would suggest asking this on the XenServer mailing lists (xs-devel@lists.xenserver.org) Roger. From owner-freebsd-xen@FreeBSD.ORG Thu Oct 9 18:41:18 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4435A74C for ; Thu, 9 Oct 2014 18:41:18 +0000 (UTC) Received: from ezwind.net (bobby.ezwind.net [199.188.211.146]) by mx1.freebsd.org (Postfix) with ESMTP id 187D0924 for ; Thu, 9 Oct 2014 18:41:16 +0000 (UTC) Received: from jayPC by ezwind.net (MDaemon PRO v9.6.5) with ESMTP id 36-md50000166497.msg for ; Thu, 09 Oct 2014 13:41:15 -0500 X-Spam-Processed: ezwind.net, Thu, 09 Oct 2014 13:41:15 -0500 (not processed: spam filter heuristic analysis disabled) X-Authenticated-Sender: jwest@ezwind.net X-MDRemoteIP: 97.91.122.42 X-Return-Path: prvs=1359fe52bd=jwest@ezwind.net X-Envelope-From: jwest@ezwind.net X-MDaemon-Deliver-To: freebsd-xen@freebsd.org From: "Jay West" To: =?iso-8859-1?Q?'Roger_Pau_Monn=E9'?= , References: <000001cfe3ca$8d242950$a76c7bf0$@ezwind.net> <5436CF13.4080509@citrix.com> In-Reply-To: <5436CF13.4080509@citrix.com> Subject: RE: disk loss Date: Thu, 9 Oct 2014 13:48:06 -0500 Message-ID: <000101cfe3f1$91407da0$b3c178e0$@ezwind.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQLZQaRIeWHWrqmNu5N4SPLKldtZ2QLBV7crmf9UgNA= Content-Language: en-us X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2014 18:41:18 -0000 Roger wrote.... ------ >When this happens, can you log into the affected VM (with the disk "supposedly" attached) and paste the output of `xenstore-ls -fp /local/domain/` (replace with the ID of the domain) here. Will do that later today, no problem. >You will need the xen-tools package installed in order to do that. Yep, xen-tools exists for Freebsd and it's always installed. >It looks to me like this is some kind of issue with the XenServer storage manager, so I would suggest asking this on the XenServer mailing lists (xs-devel@lists.xenserver.org) I would have thought so to (that it's a xenserver issue). However, it is very consistent that every time this has occurred (a handful of times), the windows guests and Linux guests are not affected. Only the FreeBSD guests are affected. J From owner-freebsd-xen@FreeBSD.ORG Fri Oct 10 10:54:31 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E651F7CB for ; Fri, 10 Oct 2014 10:54:31 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 8D0017F5 for ; Fri, 10 Oct 2014 10:54:31 +0000 (UTC) Received: from Mail-PC.tdx.co.uk (storm.tdx.co.uk [62.13.130.251]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id s9AAo7eG088994 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 10 Oct 2014 11:50:08 +0100 (BST) Date: Fri, 10 Oct 2014 11:50:06 +0100 From: Karl Pielorz To: Marko Lerota , FreeBSD XEN Subject: Re: Routing/NAT problem on Xenserver 6.2 with virtual firewall Message-ID: <971643C23A6DD25E72F11864@Mail-PC.tdx.co.uk> In-Reply-To: <86fvfxove9.fsf@arch.perpetuum.hr> References: <86k359p1qm.fsf@arch.perpetuum.hr> <9864A2A7BE97EB706ED0FC04@Mail-PC.tdx.co.uk> <86fvfxove9.fsf@arch.perpetuum.hr> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Oct 2014 10:54:32 -0000 --On 12 September 2014 14:50 +0200 Marko Lerota wrote: > Thanks Karl. It may be something that is 'easy' to fix. ICMP traffic > already goes through. So why not just add TCP/UDP in the code :-) > It would be a shame that I have to use Iptables again. Any of the > free DEVs are listening? I'm buying a beer :-). Replying late as I've just walked into this issue, yet again - which prompts me to ask: - Is there any way of having the FreeBSD virtual machine use the Xen disk services, but the legacy 'realtek' network card? I had asked about this a while ago - I think someone said it would be 'very tricky' - and much later someone replied that it might be possible... That would hopefully give us some agility back to these machines - which at the moment are languishing on a separate pool - running pure HVM, which is a real pain for backend storage changes / upgrades :( Maybe it's time for beer++? :) -Karl