From owner-freebsd-xen@freebsd.org Mon Jul 25 13:59:28 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DF4DBA4561 for ; Mon, 25 Jul 2016 13:59:28 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB81B12C2 for ; Mon, 25 Jul 2016 13:59:27 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.15.2/8.15.2) with ESMTPSA id u6PDxBAO026115 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 25 Jul 2016 14:59:12 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) Date: Mon, 25 Jul 2016 14:59:02 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , "Hoyer-Reuther, Christian" cc: freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: In-Reply-To: <20160722115542.dopzb63dgkilqall@mac> References: <41E487BC91654544B2B8F31096F2D9D4D1514D1D8E@ex1> <20160714103016.4hgfzsjgkkgtkkgg@mac> <41E487BC91654544B2B8F31096F2D9D4D1514D1E88@ex1> <20160720093111.mpmp27wol7j3ge3d@mac> <41E487BC91654544B2B8F31096F2D9D4D1516490E9@ex1> <20160722115542.dopzb63dgkilqall@mac> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 13:59:28 -0000 --On 22 July 2016 13:55 +0200 Roger Pau Monn=C3=A9 = wrote: > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 > consecutive times without seeing any issues (or freezes), although this > was with OSS Xen and without xe-guest-utilities. Karl, have you tested > HEAD recently? Ok, I have tested this with r303286 - it seems to work OK. The hosts gain=20 no time that I can see while migrating, and NTP stays happy. I did get a panic after about 40 migrations - but that seems to be some=20 network issue or something... ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic = / kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram=20 / kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall) I don't have a crashdump (failed). I did get a backtrace, for what it's=20 worth. I'm running the test again now (in case it panics again - I'll try harder=20 to get a dump just in case). -Karl From owner-freebsd-xen@freebsd.org Mon Jul 25 14:43:53 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 026A6BA32DE for ; Mon, 25 Jul 2016 14:43:53 +0000 (UTC) (envelope-from prvs=00729d017=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A9E07130E for ; Mon, 25 Jul 2016 14:43:52 +0000 (UTC) (envelope-from prvs=00729d017=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.28,419,1464652800"; d="scan'208";a="375233208" Date: Mon, 25 Jul 2016 16:43:43 +0200 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Karl Pielorz CC: "Hoyer-Reuther, Christian" , , Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <20160725144314.yhggviqhsqzgux2w@mac> References: <41E487BC91654544B2B8F31096F2D9D4D1514D1D8E@ex1> <20160714103016.4hgfzsjgkkgtkkgg@mac> <41E487BC91654544B2B8F31096F2D9D4D1514D1E88@ex1> <20160720093111.mpmp27wol7j3ge3d@mac> <41E487BC91654544B2B8F31096F2D9D4D1516490E9@ex1> <20160722115542.dopzb63dgkilqall@mac> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.6.2-neo (2016-06-11) X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 14:43:53 -0000 Adding Wei to the Cc list since he added the multiqueue functionality. On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote: > > --On 22 July 2016 13:55 +0200 Roger Pau Monné wrote: > > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 > > consecutive times without seeing any issues (or freezes), although this > > was with OSS Xen and without xe-guest-utilities. Karl, have you tested > > HEAD recently? > > Ok, I have tested this with r303286 - it seems to work OK. The hosts gain no > time that I can see while migrating, and NTP stays happy. > > I did get a panic after about 40 migrations - but that seems to be some > network issue or something... > > ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic / > kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram / > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall) I haven't been able to reproduce this, but I think it's possible that if you migrate an active netfront xn_txq_mq_start might be called during the migration, just in the middle of the setup_device reconfiguation (while info->num_queues is 0). Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order to notify the net subsystem that the queues are full, so no further calls to xn_txq_mq_start happen until the resume has finished, do you agree? Roger. From owner-freebsd-xen@freebsd.org Mon Jul 25 17:27:44 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F2E8BA3789 for ; Mon, 25 Jul 2016 17:27:44 +0000 (UTC) (envelope-from prvs=0073ad739=wei.liu2@citrix.com) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D58FF1705 for ; Mon, 25 Jul 2016 17:27:43 +0000 (UTC) (envelope-from prvs=0073ad739=wei.liu2@citrix.com) X-IronPort-AV: E=Sophos;i="5.28,420,1464652800"; d="scan'208";a="368256952" Date: Mon, 25 Jul 2016 16:37:14 +0100 From: Wei Liu To: Roger Pau =?iso-8859-1?Q?Monn=E9?= CC: Karl Pielorz , "Hoyer-Reuther, Christian" , , Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <20160725153714.GW27082@citrix.com> References: <41E487BC91654544B2B8F31096F2D9D4D1514D1D8E@ex1> <20160714103016.4hgfzsjgkkgtkkgg@mac> <41E487BC91654544B2B8F31096F2D9D4D1514D1E88@ex1> <20160720093111.mpmp27wol7j3ge3d@mac> <41E487BC91654544B2B8F31096F2D9D4D1516490E9@ex1> <20160722115542.dopzb63dgkilqall@mac> <20160725144314.yhggviqhsqzgux2w@mac> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20160725144314.yhggviqhsqzgux2w@mac> User-Agent: Mutt/1.5.23 (2014-03-12) X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jul 2016 17:27:44 -0000 On Mon, Jul 25, 2016 at 04:43:43PM +0200, Roger Pau Monné wrote: > Adding Wei to the Cc list since he added the multiqueue functionality. > > On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote: > > > > --On 22 July 2016 13:55 +0200 Roger Pau Monné wrote: > > > > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 > > > consecutive times without seeing any issues (or freezes), although this > > > was with OSS Xen and without xe-guest-utilities. Karl, have you tested > > > HEAD recently? > > > > Ok, I have tested this with r303286 - it seems to work OK. The hosts gain no > > time that I can see while migrating, and NTP stays happy. > > > > I did get a panic after about 40 migrations - but that seems to be some > > network issue or something... > > > > ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic / > > kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram / > > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall) > > I haven't been able to reproduce this, but I think it's possible that if you > migrate an active netfront xn_txq_mq_start might be called during the > migration, just in the middle of the setup_device reconfiguation (while > info->num_queues is 0). > > Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order to > notify the net subsystem that the queues are full, so no further calls to > xn_txq_mq_start happen until the resume has finished, do you agree? > Perhaps clear IFF_DRV_RUNNING and only set it when the device is ready? Looking at the manpage is seems more appropriate to me semantically. Wei. > Roger. From owner-freebsd-xen@freebsd.org Fri Jul 29 08:29:14 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4861FBA88D4 for ; Fri, 29 Jul 2016 08:29:14 +0000 (UTC) (envelope-from prvs=011b0443a=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D07691A16 for ; Fri, 29 Jul 2016 08:29:13 +0000 (UTC) (envelope-from prvs=011b0443a=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.28,438,1464652800"; d="scan'208";a="376245728" Date: Fri, 29 Jul 2016 10:29:05 +0200 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Wei Liu CC: Karl Pielorz , "Hoyer-Reuther, Christian" , Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: <20160729082905.46js7o3zp6iwuibd@mac> References: <41E487BC91654544B2B8F31096F2D9D4D1514D1D8E@ex1> <20160714103016.4hgfzsjgkkgtkkgg@mac> <41E487BC91654544B2B8F31096F2D9D4D1514D1E88@ex1> <20160720093111.mpmp27wol7j3ge3d@mac> <41E487BC91654544B2B8F31096F2D9D4D1516490E9@ex1> <20160722115542.dopzb63dgkilqall@mac> <20160725144314.yhggviqhsqzgux2w@mac> <20160725153714.GW27082@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20160725153714.GW27082@citrix.com> User-Agent: Mutt/1.6.2-neo (2016-06-11) X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 08:29:14 -0000 On Mon, Jul 25, 2016 at 04:37:14PM +0100, Wei Liu wrote: > On Mon, Jul 25, 2016 at 04:43:43PM +0200, Roger Pau Monné wrote: > > Adding Wei to the Cc list since he added the multiqueue functionality. > > > > On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote: > > > > > > --On 22 July 2016 13:55 +0200 Roger Pau Monné wrote: > > > > > > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 > > > > consecutive times without seeing any issues (or freezes), although this > > > > was with OSS Xen and without xe-guest-utilities. Karl, have you tested > > > > HEAD recently? > > > > > > Ok, I have tested this with r303286 - it seems to work OK. The hosts gain no > > > time that I can see while migrating, and NTP stays happy. > > > > > > I did get a panic after about 40 migrations - but that seems to be some > > > network issue or something... > > > > > > ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic / > > > kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram / > > > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall) > > > > I haven't been able to reproduce this, but I think it's possible that if you > > migrate an active netfront xn_txq_mq_start might be called during the > > migration, just in the middle of the setup_device reconfiguation (while > > info->num_queues is 0). > > > > Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order to > > notify the net subsystem that the queues are full, so no further calls to > > xn_txq_mq_start happen until the resume has finished, do you agree? > > > > Perhaps clear IFF_DRV_RUNNING and only set it when the device is ready? > Looking at the manpage is seems more appropriate to me semantically. Hello Karl and Christian, I have the following patches that solve all the issues I've seen with live migration, with those I've been able to migrate a VM > 100 times without seeing any issues. Could you give them a try? BTW, I haven't been able to reproduce Karl's crash ("called with 0 available queues"), but I've added a condition that should prevent it from triggering anyway. Patches are here: https://reviews.freebsd.org/D7349 https://reviews.freebsd.org/D7362 https://reviews.freebsd.org/D7363 It doesn't really matter in which order you apply them as long as both 3 are applied. Ideally I would like to commit them on Monday, so that I can MFC them to stable/11 before the releng/11 branch, could you please provide some feedback before then? Thanks, Roger. From owner-freebsd-xen@freebsd.org Fri Jul 29 14:58:31 2016 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D41CBA78ED for ; Fri, 29 Jul 2016 14:58:31 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.krpservers.com", Issuer "RapidSSL SHA256 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 14D4A1C01 for ; Fri, 29 Jul 2016 14:58:30 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213] (may be forged)) (authenticated bits=0) by smtp.krpservers.com (8.15.2/8.15.2) with ESMTPSA id u6TEwGQX014598 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 29 Jul 2016 15:58:17 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) Date: Fri, 29 Jul 2016 15:57:55 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , Wei Liu cc: "Hoyer-Reuther, Christian" , freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? Message-ID: In-Reply-To: <20160729082905.46js7o3zp6iwuibd@mac> References: <41E487BC91654544B2B8F31096F2D9D4D1514D1D8E@ex1> <20160714103016.4hgfzsjgkkgtkkgg@mac> <41E487BC91654544B2B8F31096F2D9D4D1514D1E88@ex1> <20160720093111.mpmp27wol7j3ge3d@mac> <41E487BC91654544B2B8F31096F2D9D4D1516490E9@ex1> <20160722115542.dopzb63dgkilqall@mac> <20160725144314.yhggviqhsqzgux2w@mac> <20160725153714.GW27082@citrix.com> <20160729082905.46js7o3zp6iwuibd@mac> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jul 2016 14:58:31 -0000 --On 29 July 2016 10:29 +0200 Roger Pau Monn=C3=A9 = wrote: > Hello Karl and Christian, I have the following patches that solve all the > issues I've seen with live migration, with those I've been able to > migrate a VM > 100 times without seeing any issues. Could you give them > a try? > > BTW, I haven't been able to reproduce Karl's crash ("called with 0 > available queues"), but I've added a condition that should prevent it > from triggering anyway. Patches are here: > > https://reviews.freebsd.org/D7349 > https://reviews.freebsd.org/D7362 > https://reviews.freebsd.org/D7363 > > It doesn't really matter in which order you apply them as long as both 3 > are applied. Ideally I would like to commit them on Monday, so that I > can MFC them to stable/11 before the releng/11 branch, could you please > provide some feedback before then? Patched, and have been running migrations back & forth between two pool=20 members all day (must have done ~100), time has stayed in sync - and I've=20 not experienced any panics. This is on the same VM as before - I've also tried leaving more heavy=20 network processes running in the background, and still seems Ok. Regards, -Karl