From owner-freebsd-virtualization@freebsd.org Mon Feb 19 10:07:19 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17CAAF15C53; Mon, 19 Feb 2018 10:07:19 +0000 (UTC) (envelope-from prvs=5814e79b2=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 584346B31C; Mon, 19 Feb 2018 10:07:18 +0000 (UTC) (envelope-from prvs=5814e79b2=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.46,534,1511827200"; d="scan'208";a="68150322" Date: Mon, 19 Feb 2018 10:05:58 +0000 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Laurence Pawling CC: "freebsd-xen@freebsd.org" , "freebsd-virtualization@freebsd.org" , "freebsd-net@freebsd.org" , David King , Vlad Galu Subject: Re: multi-vCPU networking issues as client OS under Xen Message-ID: <20180219100558.adgb6m5ukdfvxehp@MacBook-Pro-de-Roger.local> References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: NeoMutt/20171208 X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL02.citrite.net (10.69.22.126) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Feb 2018 10:07:19 -0000 On Mon, Feb 19, 2018 at 09:58:30AM +0000, Laurence Pawling via freebsd-xen wrote: > Hi all, > > > > I’m wondering if anyone here has seen this issue before, I’ve spent the last couple of days troubleshooting: > > > > Platform: > > Host: XenServer 7.0 running on 2 x E2660-v4, 256GB RAM > > Server VM: FreeBSD 11 (tested on 11.0-p15 and 11.1-p6), 2GB RAM (also tested with 32GB RAM), 1x50GB HDD, 1 x NIC, 2 or more vCPUs in any combination (2 sockets x 1 core, 1 socket x 2 cores, …) > > Client VM: FreeBSD 11, any configuration of vCPUs, RAM and HDD. > > > > Behaviour: > > Sporadic interruption of TCP sessions when utilising the above machine as a “server” with “clients” connecting. Looking into the communication with pcap/Wireshark, you see a TCP Dup Ack sent from both ends, followed by the client sending an RST packet, terminating the TCP session. We have also seen evidence of the client sending a Keepalive packet, which is ACK’d by the server before the RST is sent from the client end. > > > > To recreate: > > On the above VM, perform a vanilla install of nginx: > > pkg install nginx > > service nginx onestart > > Then on a client VM (currently only tested with FreeBSD), run the following (or similar): > > for i in {1..10000}; do if [ $(curl -s -o /dev/null -w "%{http_code}" http://10.2.122.71) != 200 ] ; then echo "error"; fi; done > > When vCPUs=1 on the server, I get no errors, when vCPUs>1 I get errors reported. The frequency of errors *seems* to be proportional to the number of vCPUs, but they are sporadic with no clear periodicity or pattern, so that is just anecdotal. Also, the problem seems by far the most prevalent when communicating between two VMs on the same host, in the same VLAN. Xen still sends packets via the switch rather than bridging internally between the interfaces. When using >1 vCPUs can you set hw.xn.num_queues=1 on /boot/loader.conf and try to reproduce the issue? I'm afraid this is rather related to multiqueue (which is only used if >1 vCPUs). Thanks, Roger.