From owner-freebsd-xen@FreeBSD.ORG Wed Mar 25 07:30:26 2015 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 88C07C97 for ; Wed, 25 Mar 2015 07:30:26 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 558F4C6 for ; Wed, 25 Mar 2015 07:30:26 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t2P7UQuf096148 for ; Wed, 25 Mar 2015 07:30:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-xen@FreeBSD.org Subject: [Bug 195978] Add vlan support to xen netback Date: Wed, 25 Mar 2015 07:30:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: araujo@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-xen@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2015 07:30:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195978 Marcelo Araujo changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |araujo@FreeBSD.org --- Comment #1 from Marcelo Araujo --- (In reply to Grischa Zengel from comment #0) Hello Grischa, I made few tests using FreeBSD HEAD(r280410) as dom0 and two domU with FreeBSD 10.1-RELEASE. The xen version is 4.6-unstable. I successfully setup a vlan(4) interface without any problem and without the patch you mentioned, here is the output: Dom0: root@:/z/src/sys/dev/xen/netfront # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2047 8 r----- 373.0 FreeBSDPVHVM2 3 512 2 -b---- 20.2 FreeBSDPVHVM 4 512 2 -b---- 29.0 FreeBSDPVHVM: root@pvhvm:~ # ifconfig xn0.10 xn0.10: flags=8843 metric 0 mtu 1496 ether 00:16:3e:50:e2:8d inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet manual status: active vlan: 10 parent interface: xn0 FreeBSDPVHVM2: root@pvhvm2:~ # ifconfig xn0.10 xn0.10: flags=8843 metric 0 mtu 1496 ether 00:16:3e:58:40:14 inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet manual status: active vlan: 10 parent interface: xn0 root@pvhvm:~ # ping 192.168.1.2 PING 192.168.1.2 (192.168.1.2): 56 data bytes 64 bytes from 192.168.1.2: icmp_seq=0 ttl=64 time=0.178 ms 64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.171 ms 64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=0.147 ms root@pvhvm2:~ # ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.134 ms 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.118 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.184 ms I'm wondering, if you still have this problem, and if you can give a try with FreeBSD HEAD. Best Regards, -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-xen@FreeBSD.ORG Sat Mar 28 03:57:22 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD9459B9 for ; Sat, 28 Mar 2015 03:57:22 +0000 (UTC) Received: from os-mail-5.tamu.edu (os-mail-5.tamu.edu [165.91.22.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp-relay.tamu.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C3C5689 for ; Sat, 28 Mar 2015 03:57:21 +0000 (UTC) X-TAMU-Auth-ID: None X-TAMU-SenderIP: 165.91.22.240 Received: from exchange.tamu.edu (exch-2p-snat-pool.tamu.edu [165.91.22.240]) by os-mail-5.itio.tamu.edu (8.14.7/8.14.7) with ESMTP id t2RNDOMF004462 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Fri, 27 Mar 2015 18:13:24 -0500 Received: from MB03.ads.tamu.edu ([169.254.3.105]) by caex02.ads.tamu.edu ([128.194.147.21]) with mapi id 14.03.0195.001; Fri, 27 Mar 2015 18:13:23 -0500 From: Andrew Daugherity To: "freebsd-xen@freebsd.org" Subject: Poor performance with FreeBSD 10.1 under Xen 4.2 Thread-Topic: Poor performance with FreeBSD 10.1 under Xen 4.2 Thread-Index: AQHQaOOfinlni718o0SDQUrW328qGA== Date: Fri, 27 Mar 2015 23:13:23 +0000 Message-ID: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.194.89.247] Content-Type: text/plain; charset="us-ascii" Content-ID: <9861A56D8560CC45B9BB8229462B4A98@exchange.tamu.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.13.68, 1.0.33, 0.0.0000 definitions=2015-03-27_08:2015-03-27,2015-03-27,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 kscore.is_bulkscore=6.11347639178916e-11 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.999586866343301 suspectscore=0 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 rbsscore=0.999586866343301 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=0 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.999586866343301 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1503270226 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 03:57:22 -0000 Summary: FreeBSD 10.1/amd64 under Xen 4.2.5 is much slower than FreeBSD 9.3= on the same environment, especially at fork() I recently installed a FreeBSD-10.1 VM under Xen, and was pleased to see th= e XENHVM stuff is now integrated into GENERIC. However, the system seemed = a little slow and lacking in "snappiness" -- the first fetch/extraction of = portsnap was particularly bad, taking at least 20 minutes. It had been a w= hile since I'd done that (as opposed to 'portsnap fetch update') so I wasn'= t sure how abnormal that was, but then I noticed building stuff from ports,= especially stuff using libtool, like security/sssd, was extremely slow com= pared to physical hardware, so I tested a 9.3 VM, which was much faster. Importantly, it was not a typical case of a slow/overloaded CPU but more li= ke slow context switching/forking. I would see high (40%) system CPU perce= ntage but low user, and usually the process at the top of the list was sh. = It would take a long time between compiling files but when cc finally ran = it was quite fast, compiling each file in a second or two. The system was = not swapping and iostat (also xentop on the host) showed minimal I/O load. Tracing the sh process (which was libtool-related) with truss, I would see = it do some stuff, fork, wait several seconds, then do some more stuff, rins= e and repeat. Using 'truss -f' to follow the child processes, there was a = noticeable delay associated with each fork() call. This led me to do some benchmarking. I found a fork() benchmark at [1] and= ran it on various systems. Notably, on FreeBSD 10.1 (also 10.0) under Xen= , it was reasonably fast shortly after bootup (though still slower than 9.3= ), but would get slower on repeated runs, and significantly slower after co= mpiling some ports. It would also run slowly if the system had booted and = then sat idle for a while. The speed was inconsistent, as occasionally afte= r a period of idleness it would run somewhat faster again without rebooting= ; also configure and compilation times of sssd were inconsistent, but gener= ally "slow", sometimes drastically so. FreeBSD 9.3 (with "xenhvm_load=3D"YES" in loader.conf) on the same Xen host= does not have this problem -- it fork()s more quickly and consistently; Fr= eeBSD 10.1 on KVM (unfortunately not on the same hardware) also appears nor= mal, as does 8.4 on (different but similar vintage) physical hardware, and = a Linux VM on the same Xen host. Using one or two virtual CPUs does not ma= ke much difference, and the host machine is otherwise idle, so it does not = appear to be an SMP issue. I was using ZFS, but I have ruled that out as a= factor, as the problem occurs even without zfs.ko loaded (/ is ufs). Vary= ing the memory between 1 and 8 GB did not seem to affect anything either. = I also built a "NOHVM" 10.1 kernel to see if the Xen drivers were at issue,= but that did not help (it was actually a bit slower), so it appears to be = something deeper in the kernel or scheduler. The Xen host is running Xen 4.2.5_02-0.7.1 with SLES 11 SP3 as the Dom0, on= a Dell 2950 with 8 physical CPU cores (dual socket, quad-core Xeon E5420).= I have not experienced performance problems with any other guest OS. As FreeBSD 9.3 runs fine, I am using that for my FreeBSD VMs for now, but h= opefully 10.x can be fixed before 9-STABLE goes EOL! Following are the VM = config, dmesg, and some benchmarks. -Andrew [1] https://github.com/mondalaci/fork-benchmark Xen DomU config: =3D=3D=3D=3D=3D=3D=3D=3D name=3D"fbsd10" description=3D"FreeBSD 10.1 - testing" uuid=3D"ed88195c-dee4-0e44-5943-3deceac8a56c" #memory=3D4096 memory=3D1024 maxmem=3D1024 vcpus=3D2 on_poweroff=3D"destroy" on_reboot=3D"restart" on_crash=3D"preserve" localtime=3D0 keymap=3D"en-us" builder=3D"hvm" device_model=3D"/usr/lib/xen/bin/qemu-dm" kernel=3D"/usr/lib/xen/boot/hvmloader" boot=3D"c" disk=3D[ 'phy:/dev/xc-test/fbsd10,hda,w', 'file:/root/FreeBSD-10.1-RELEASE-= amd64-dvd1.iso,hdc:cdrom,r', ] vif=3D[ 'mac=3D00:16:3e:3a:57:7a,bridge=3Dbr0,type=3Dnetfront', ] stdvga=3D0 vnc=3D1 vncunused=3D1 viridian=3D0 acpi=3D1 pae=3D1 serial=3D"pty" =3D=3D=3D=3D=3D=3D=3D=3D dmesg: =3D=3D=3D=3D=3D=3D=3D=3D Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 XEN: Hypervisor version 4.2 detected. CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2493.90-MHz K8-class = CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Family =3D 0x6 Model =3D 0x17= Stepping =3D 6 Features=3D0x1783fbff Features2=3D0x81282201 AMD Features=3D0x20100800 AMD Features2=3D0x1 real memory =3D 1073741824 (1024 MB) avail memory =3D 1010737152 (963 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 ioapic0: Changing APIC ID to 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-47 on motherboard kbd1 at kbdmux0 random: initialized xen_et0: on motherboard Event timer "XENTIMER" frequency 1000000000 Hz quality 950 Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) acpi0: reservation of 0, a0000 (3) failed cpu0: on acpi0 cpu1: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,= 0x376,0xc100-0xc10f at device 1.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 1.3 (no driver attached) vgapci0: mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3= 000fff at device 2.0 on pci0 vgapci0: Boot video device xenpci0: port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff= irq 28 at device 3.0 on pci0 xenstore0: on xenpci0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (9600,n,8,1) ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 fdc0: No FDOUT register! Timecounters tick every 10.000 msec xctrl0: on xenstore0 xenbusb_front0: on xenstore0 cd0 at ata1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: Serial Number QM00003 cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: cd present [1262221 x 2048 byte records] xbd0: 6144MB at device/vbd/768 on xenbusb_front0 xbd0: attaching as ada0 xbd0: features: flush, write_barrier xbd0: synchronize cache commands enabled. xn0: at device/vif/0 on xenbusb_front0 xn0: Ethernet address: 00:16:3e:3a:57:7a xenbusb_back0: on xenstore0 xn0: backend features: feature-sg feature-gso-tcp4 random: unblocking device. SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/ada0p2 [rw]... xn0: 2 link states coalesced =3D=3D=3D=3D=3D=3D=3D=3D "NOHVM" kernel config (not the dmesg above, but presented for completeness)= : =3D=3D=3D=3D=3D=3D=3D=3D include GENERIC ident NOHVM # NOTE: XENHVM depends on xenpci. They must be added or removed together. nooptions XENHVM # Xen HVM kernel infrastructure nodevice xenpci # Xen HVM Hypervisor services driver =3D=3D=3D=3D=3D=3D=3D=3D Benchmarks: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Fork benchmark -- ./fork-benchmark : 10.1, 2 CPU, fresh boot: Forked, executed and destroyed 100 processes in 0.268835 seconds. Forked, executed and destroyed 1000 processes in 2.362202 seconds. Forked, executed and destroyed 1000 processes in 2.642716 seconds. Forked, executed and destroyed 10000 processes in 28.75984 seconds. Forked, executed and destroyed 10000 processes in 34.568837 seconds. Forked, executed and destroyed 10000 processes in 52.69006 seconds. Forked, executed and destroyed 10000 processes in 53.41585 seconds. 10.1, 1 CPU, after compiling sssd: Forked, executed and destroyed 100 processes in 5.684971 seconds. Forked, executed and destroyed 1000 processes in 60.330680 seconds. 10.1, 2 CPU, NOHVM kernel, after compiling sssd: Forked, executed and destroyed 5000 processes in 102.849662 seconds. Forked, executed and destroyed 5000 processes in 107.160831 seconds. Forked, executed and destroyed 100 processes in 2.524160 seconds. Forked, executed and destroyed 1000 processes in 19.592753 seconds. 9.3, 1 CPU: Forked, executed and destroyed 5000 processes in 8.416964 seconds. 9.3, 2 CPU: 1: Forked, executed and destroyed 5000 processes in 9.951971 seconds. 2: Forked, executed and destroyed 5000 processes in 10.185864 seconds. 3: Forked, executed and destroyed 5000 processes in 10.124263 seconds. (remains consistent) Compilation times -- cd /usr/ports/security/sssd; make clean; time make con= figure; time make build configure: 9.3, 1 CPU: 22.804u 10.764s 0:40.19 83.5% 1400+2497k 816+7885io 456pf+0w 9.3, 2 CPU: 25.732u 14.651s 0:42.38 95.2% 1326+2432k 164+7885io 30pf+0w 10.1, 1 CPU: 148.992u 68.372s 3:38.52 99.4% 2325+197k 0+294io 3pf+0w 10.1, 2 CPU: 1.156u 29.289s 1:02.47 96.7% 4602+225k 774+300io 654pf+0w (again): 35.229u 21.117s 0:49.30 114.2% 4667+221k 0+291io 0pf+0w 10.1 NOHVM: 80.236u 51.313s 1:51.45 118.0% 2930+200k 0+296io 30pf+0w build: 9.3, 1 CPU: 233.998u 145.352s 6:22.51 99.1% 1360+2777k 287+3966io 32pf+0w 9.3, 2 CPU: 280.641u 230.728s 4:24.23 193.5% 1157+2675k 0+3968io 0pf+0w 10.1, 1 CPU: 3199.849u 764.871s 1:06:26.72 99.4% 753+182k 203+28io 86pf+0w 10.1, 2 CPU: 744.318u 549.327s 11:02.38 195.3% 2388+193k 235+28io 86pf+0w (again): 1072.863u 747.565s 15:30.05 195.7% 2119+192k 3+29io 0pf+0w 10.1 NOHVM: 1173.692u 823.116s 17:06.46 194.5% 1725+188k 0+28io 0pf+0w Note the 10.1/1 CPU build took over an hour! I'm fairly certain I had a 10= .1/2 CPU build also take around an hour, but I didn't manage to capture it = with time(1).= From owner-freebsd-xen@FreeBSD.ORG Sat Mar 28 08:56:23 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9DAC290 for ; Sat, 28 Mar 2015 08:56:23 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id B341E794 for ; Sat, 28 Mar 2015 08:56:23 +0000 (UTC) Received: from [10.12.30.100] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id t2S8reeh009038 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 28 Mar 2015 08:53:41 GMT Date: Sat, 28 Mar 2015 08:53:40 +0000 From: Karl Pielorz To: Andrew Daugherity , freebsd-xen@freebsd.org Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 Message-ID: <1E351BDCCC425075995A562C@Karls-Mac-mini.local> In-Reply-To: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> X-Mailer: Mulberry/4.0.8 (Mac OS X) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 08:56:24 -0000 --On 27 March 2015 23:13:23 +0000 Andrew Daugherity wrote: > Summary: FreeBSD 10.1/amd64 under Xen 4.2.5 is much slower than FreeBSD > 9.3 on the same environment, especially at fork() > > [snip] Hi, We've got a number of 9.3, 10.0 and growing number of 10.1 boxes running under XenServer 6.5 and OnApp (using Xen) here - and I've not noticed any issues with them. As you've carefully provided all the info - I'll give the same bench mark a run on some of our systems here next week - and post the results. Obviously it's not an 'apples for apples' comparison - XenServer 6.5, from memory uses Xen 4.4. I think the OnApp system we're using Xen 3.4 If nothing else it'd be good to know we really don't have an issue rather than one we've not noticed (otherwise this might turn into a '/me too' post :) -Karl From owner-freebsd-xen@FreeBSD.ORG Sat Mar 28 13:16:19 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D1734CA; Sat, 28 Mar 2015 13:16:19 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Cybertrust Public SureServer SV CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D1F741A2; Sat, 28 Mar 2015 13:16:18 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.11,484,1422921600"; d="scan'208";a="247515578" Message-ID: <5516A998.10206@citrix.com> Date: Sat, 28 Mar 2015 14:16:08 +0100 From: =?windows-1252?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Andrew Daugherity , "freebsd-xen@freebsd.org" Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> In-Reply-To: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 13:16:19 -0000 Hello, El 28/03/15 a les 0.13, Andrew Daugherity ha escrit: > Summary: FreeBSD 10.1/amd64 under Xen 4.2.5 is much slower than FreeBSD 9.3 on the same environment, especially at fork() > > > I recently installed a FreeBSD-10.1 VM under Xen, and was pleased to see the XENHVM stuff is now integrated into GENERIC. However, the system seemed a little slow and lacking in "snappiness" -- the first fetch/extraction of portsnap was particularly bad, taking at least 20 minutes. It had been a while since I'd done that (as opposed to 'portsnap fetch update') so I wasn't sure how abnormal that was, but then I noticed building stuff from ports, especially stuff using libtool, like security/sssd, was extremely slow compared to physical hardware, so I tested a 9.3 VM, which was much faster. > > Importantly, it was not a typical case of a slow/overloaded CPU but more like slow context switching/forking. I would see high (40%) system CPU percentage but low user, and usually the process at the top of the list was sh. It would take a long time between compiling files but when cc finally ran it was quite fast, compiling each file in a second or two. The system was not swapping and iostat (also xentop on the host) showed minimal I/O load. > > Tracing the sh process (which was libtool-related) with truss, I would see it do some stuff, fork, wait several seconds, then do some more stuff, rinse and repeat. Using 'truss -f' to follow the child processes, there was a noticeable delay associated with each fork() call. > > This led me to do some benchmarking. I found a fork() benchmark at [1] and ran it on various systems. Notably, on FreeBSD 10.1 (also 10.0) under Xen, it was reasonably fast shortly after bootup (though still slower than 9.3), but would get slower on repeated runs, and significantly slower after compiling some ports. It would also run slowly if the system had booted and then sat idle for a while. The speed was inconsistent, as occasionally after a period of idleness it would run somewhat faster again without rebooting; also configure and compilation times of sssd were inconsistent, but generally "slow", sometimes drastically so. > > FreeBSD 9.3 (with "xenhvm_load="YES" in loader.conf) on the same Xen host does not have this problem -- it fork()s more quickly and consistently; FreeBSD 10.1 on KVM (unfortunately not on the same hardware) also appears normal, as does 8.4 on (different but similar vintage) physical hardware, and a Linux VM on the same Xen host. Using one or two virtual CPUs does not make much difference, and the host machine is otherwise idle, so it does not appear to be an SMP issue. I was using ZFS, but I have ruled that out as a factor, as the problem occurs even without zfs.ko loaded (/ is ufs). Varying the memory between 1 and 8 GB did not seem to affect anything either. I also built a "NOHVM" 10.1 kernel to see if the Xen drivers were at issue, but that did not help (it was actually a bit slower), so it appears to be something deeper in the kernel or scheduler. > > The Xen host is running Xen 4.2.5_02-0.7.1 with SLES 11 SP3 as the Dom0, on a Dell 2950 with 8 physical CPU cores (dual socket, quad-core Xeon E5420). I have not experienced performance problems with any other guest OS. > > As FreeBSD 9.3 runs fine, I am using that for my FreeBSD VMs for now, but hopefully 10.x can be fixed before 9-STABLE goes EOL! Following are the VM config, dmesg, and some benchmarks. Thanks for the detailed description of the issue. I've reproduced your benchmarks and with the hardware I've used (Core i3-5010U) I'm not able to see this performance issue, below are the figures in my case: FreeBSD 9.3 after boot fork 1000 0.538132 FreeBSD 10.1 after boot fork 1000 0.514381 FreeBSD 9.3 build sssd 2098.66 real 1726.37 user 446.76 sys FreeBSD 10.1 build sssd 1247.38 real 1180.35 user 294.41 sys FreeBSD 9.3 after sssd fork 1000 0.543631 FreeBSD 10.1 after sssd fork 1000 0.538207 I'm Ccing feld because IIRC he found something similar on one of his boxes, that also had VTx but no EPT (just like yours). Would it be possible for you to try the same set of tests on a different hardware? Also, if even FreeBSD 10.1 compiled without XENHVM shows this issue it means there's something in the generic code that doesn't work well when running virtualized on this specific hardware, but I'm afraid figuring it out is not trivial. One place to start would be asking on freebsd-hackers and freebsd-virt. Roger. From owner-freebsd-xen@FreeBSD.ORG Sat Mar 28 13:51:11 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3775494C for ; Sat, 28 Mar 2015 13:51:11 +0000 (UTC) Received: from new2-smtp.messagingengine.com (new2-smtp.messagingengine.com [66.111.4.224]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 044C36B4 for ; Sat, 28 Mar 2015 13:51:10 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 500F2E4D for ; Sat, 28 Mar 2015 09:51:00 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute3.internal (MEProxy); Sat, 28 Mar 2015 09:51:03 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=DiOrV+Ha7yPqMWs l0XWA3szJtEA=; b=E+g0UoJlenWS7WNNnPM+EcTT0si3sVasWDdzwd7x3zD96bt ydlvY3mCw4mA9uJn8jp/mtOn3dkbGFkkJyQsDrSROMPGfk69PWNG7unDwiBT7Ogy PKv/5oSGUWLWaVU+IJGYszj28oHMS/u8QBA/jET6B9CdySv+uyNCGr7CeX2U= Received: by web3.nyi.internal (Postfix, from userid 99) id EFA2F110D62; Sat, 28 Mar 2015 09:51:02 -0400 (EDT) Message-Id: <1427550662.3089969.246390749.09DD17FB@webmail.messagingengine.com> X-Sasl-Enc: Mk9UaVWo/DT12pRFjTPzKJktK9A9seA1xXkBv1fIfld2 1427550662 From: Mark Felder To: =?ISO-8859-1?Q?Roger=20Pau=20Monn=E9?= , Andrew Daugherity , freebsd-xen@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: MessagingEngine.com Webmail Interface - ajax-f437beda Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 Date: Sat, 28 Mar 2015 08:51:02 -0500 In-Reply-To: <5516A998.10206@citrix.com> References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> <5516A998.10206@citrix.com> X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 13:51:11 -0000 On Sat, Mar 28, 2015, at 08:16, Roger Pau Monn=E9 wrote: >=20 > I'm Ccing feld because IIRC he found something similar on one of his > boxes, that also had VTx but no EPT (just like yours). Would it be > possible for you to try the same set of tests on a different hardware? >=20 Sadly I no longer have access to that hardware, or any FreeBSD on Xen right now.