From owner-freebsd-xen@FreeBSD.ORG Thu May 23 09:06:23 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1FEE198F; Thu, 23 May 2013 09:06:23 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [46.33.159.39]) by mx1.freebsd.org (Postfix) with ESMTP id 07866E91; Thu, 23 May 2013 09:06:21 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.87,726,1363132800"; d="scan'208";a="4887773" Received: from lonpex01cl01.citrite.net ([10.30.203.101]) by LONPIPO01.EU.CITRIX.COM with ESMTP/TLS/AES128-SHA; 23 May 2013 09:06:19 +0000 Received: from [192.168.1.30] (10.30.203.1) by LONPEX01CL01.citrite.net (10.30.203.101) with Microsoft SMTP Server id 14.2.342.3; Thu, 23 May 2013 10:06:19 +0100 Message-ID: <519DDC0A.9000201@citrix.com> Date: Thu, 23 May 2013 11:06:18 +0200 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Colin Percival Subject: Re: FreeBSD PVHVM call for testing References: <519131D8.9010307@citrix.com> <51952BAE.6010609@freebsd.org> <51957D42.9060801@citrix.com> <51959ED9.6040405@freebsd.org> <51974EC9.9030204@citrix.com> <5197A1EA.2040404@freebsd.org> <519CAFC7.1070908@citrix.com> <519D24A9.3050407@freebsd.org> In-Reply-To: <519D24A9.3050407@freebsd.org> Content-Type: multipart/mixed; boundary="------------000309090701090501040108" X-Originating-IP: [10.30.203.1] Cc: "freebsd-xen@freebsd.org" , xen-devel , "freebsd-virtualization@freebsd.org" , "xen-users@lists.xen.org" X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 May 2013 09:06:23 -0000 --------------000309090701090501040108 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 8bit On 22/05/13 22:03, Colin Percival wrote: > On 05/22/13 04:45, Roger Pau Monné wrote: >> On 18/05/13 17:44, Colin Percival wrote: >>> That seems to work. dmesg is attached. Are there any particular tests >>> you'd like me to run? >> >> I have not tested ZFS, that might be a good one. If you are running this >> on Xen 3.4 the behaviour should be the same as without this patches, so >> there shouldn't be many differences. > > I don't use ZFS personally, so I'm not sure exactly what tests to run on it; > hopefully someone else can take care of that. > >> If you could try that on Xen 4.0 at least (if I remember correctly >> that's when the vector callback was introduced), you should see the PV >> timer getting attached, and a performance increase. > > Testing on a cr1.8xlarge EC2 instance, I get Xen 4.2, but it ends up with > a panic -- console output below. I can get a backtrace and possibly even > a dump if those would help. Hello Colin, Thanks for the test, I've been using Xen 4.2 (and 4.3) without problems so far. By looking at the Xen code, the only reason the timer setup could return -22 (EINVAL), is that we try to set the timer for a different vCPU than the one we are running on. I've been able to boot a 32 vCPU DomU on my 8way box using Xen 4.2.1 (using both qemu-xen and qemu-xen-traditional device models), so I'm unsure if this could be due to some patch Amazon applies to Xen. Could you try the following patch and post the error message? I would like to see if the cpuid reported by kdb and the vCPU that we are trying to set the timer are the same. Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #68: Wed May 22 19:00:14 CEST 2013 root@:/usr/obj/usr/src/sys/XENHVM amd64 FreeBSD clang version 3.3 (trunk 178860) 20130405 WARNING: WITNESS option enabled, expect reduced performance. XEN: Hypervisor version 4.2 detected. CPU: Intel(R) Xeon(R) CPU W3550 @ 3.07GHz (3066.83-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5 Features=0x1783fbff Features2=0x81b82201 AMD Features=0x28100800 AMD Features2=0x1 real memory = 4286578688 (4088 MB) avail memory = 3961323520 (3777 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 16 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 cpu4 (AP): APIC ID: 8 cpu5 (AP): APIC ID: 10 cpu6 (AP): APIC ID: 12 cpu7 (AP): APIC ID: 14 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 18 cpu10 (AP): APIC ID: 20 cpu11 (AP): APIC ID: 22 cpu12 (AP): APIC ID: 24 cpu13 (AP): APIC ID: 26 cpu14 (AP): APIC ID: 28 cpu15 (AP): APIC ID: 30 cpu16 (AP): APIC ID: 32 cpu17 (AP): APIC ID: 34 cpu18 (AP): APIC ID: 36 cpu19 (AP): APIC ID: 38 cpu20 (AP): APIC ID: 40 cpu21 (AP): APIC ID: 42 cpu22 (AP): APIC ID: 44 cpu23 (AP): APIC ID: 46 cpu24 (AP): APIC ID: 48 cpu25 (AP): APIC ID: 50 cpu26 (AP): APIC ID: 52 cpu27 (AP): APIC ID: 54 cpu28 (AP): APIC ID: 56 cpu29 (AP): APIC ID: 58 cpu30 (AP): APIC ID: 60 cpu31 (AP): APIC ID: 62 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-47 on motherboard kbd1 at kbdmux0 xen_et0: on motherboard Event timer "XENTIMER" frequency 1000000000 Hz quality 950 Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) acpi0: reservation of 0, a0000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 cpu24: on acpi0 cpu25: on acpi0 cpu26: on acpi0 cpu27: on acpi0 cpu28: on acpi0 cpu29: on acpi0 cpu30: on acpi0 cpu31: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 62500000 Hz quality 950 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc300-0xc30f at device 1.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 1.3 (no driver attached) vgapci0: mem 0xf0000000-0xf1ffffff,0xf3030000-0xf3030fff at device 2.0 on pci0 xenpci0: port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0 xenstore0: on xenpci0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (9600,n,8,1) ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 qpi0: on motherboard sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 fdc0: No FDOUT register! Timecounters tick every 10.000 msec xenbusb_front0: on xenstore0 xenbusb_add_device: Device device/suspend/event-channel ignored. State 6 cd0 at ata1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present xn0: at device/vif/0 on xenbusb_front0 xn0: Ethernet address: 00:16:3e:47:d4:52 xn1: at device/vif/1 on xenbusb_front0 xn1: Ethernet address: 00:16:3e:47:d4:53 xenbusb_back0: on xenstore0 xctrl0: on xenstore0 xn0: backend features: feature-sg feature-gso-tcp4 xn1: backend features: feature-sg feature-gso-tcp4 xbd0: 20480MB at device/vbd/768 on xenbusb_front0 xbd0: attaching as ad0 xbd0: disk supports cache flush using: flush SMP: AP CPU #25 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #27 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #30 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #29 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #28 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #31 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #24 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #26 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #21 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/ad0p2 [rw]... Setting hostuuid: c9230f36-1a54-489e-877c-1d15b8f463e9. Setting hostid: 0xd52252c7. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad0p2: clean, 1690868 free (11852 frags, 209877 blocks, 0.2% fragmentation) Mounting local file systems:. Writing entropy file:. xn0: link state changed to DOWN xn0: link state changed to UP Starting Network: lo0 xn0 xn1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 xn0: flags=8843 metric 0 mtu 1500 options=503 ether 00:16:3e:47:d4:52 nd6 options=29 media: Ethernet manual status: active xn1: flags=8802 metric 0 mtu 1500 options=503 ether 00:16:3e:47:d4:53 nd6 options=29 media: Ethernet manual status: active Starting devd. Starting Network: xn1. xn1: flags=8802 metric 0 mtu 1500 options=503 ether 00:16:3e:47:d4:53 nd6 options=29 media: Ethernet manual status: active Starting dhclient. DHCPREQUEST on xn0 to 255.255.255.255 port 67 DHCPACK from 192.168.1.1 bound to 192.168.1.107 -- renewal in 43200 seconds. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. Starting watchdogd. watchdogd: patting the dog: Operation not supported /etc/rc: WARNING: failed to start watchdogd No core dumps found. lock order reversal: 1st 0xffffff80f7627d80 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3070 2nd 0xfffffe0027c18000 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff8121720450 kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff8121720500 witness_checkorder() at witness_checkorder+0xc3f/frame 0xffffff8121720580 _sx_xlock() at _sx_xlock+0x75/frame 0xffffff81217205c0 ufsdirhash_add() at ufsdirhash_add+0x3b/frame 0xffffff8121720600 ufs_direnter() at ufs_direnter+0x688/frame 0xffffff81217206c0 ufs_mkdir() at ufs_mkdir+0x863/frame 0xffffff81217208c0 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xf0/frame 0xffffff81217208f0 kern_mkdirat() at kern_mkdirat+0x21a/frame 0xffffff8121720ae0 amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121720bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121720bf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092faaa, rsp = 0x7fffffffd788, rbp = 0x7fffffffdc70 --- Clearing /tmp (X related). Updating motd:. Configuring syscons: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu May 23 10:08:04 CEST 2013 FreeBSD/amd64 (Amnesiac) (ttyu0) --------------000309090701090501040108 Content-Type: text/plain; charset="UTF-8"; x-mac-type=0; x-mac-creator=0; name="timer_panic.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="timer_panic.patch" diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index 7083e46..4cb49b0 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -418,8 +418,8 @@ xentimer_et_start(struct eventtimer *et, } while (error == -ETIME); if (error) - panic("%s: Error %d setting singleshot timer to %"PRIu64"\n", - device_get_nameunit(sc->dev), error, next_time); + panic("%s: Error %d setting singleshot timer to %"PRIu64" for vCPU#%d\n", + device_get_nameunit(sc->dev), error, next_time, cpu); pcpu->timer = next_time; return (error); --------------000309090701090501040108--