Date: Mon, 20 Dec 2010 19:44:13 +1100 From: Alex <joovke@joovke.com> To: freebsd-xen@freebsd.org Subject: lockups/freezes/pauses - 8.1-release/amd64 + XEN Message-ID: <4D0F175D.8070701@joovke.com>
next in thread | raw e-mail | index | archive | help
Hi Guys,
I am having rather odd issue where the system locks up, I have purchased
a XEN based VPS and decided to go with FreeBSD as the OS. I frequently
get cases where there seem to be noticeable pauses, ie: I could be SSH'd
in and issue an "ls" command or "mc" to start midnight commander, as
soon as i press enter to run the command, there is often a good pause
for 20 seconds or so. Load averages are almost non existent because it's
a new install, almost no network traffic hitting it. if i ping the IP
address of the VPS during the pauses, there is abnormally high latency
(instead of 200ms, 1500-2000ms). A traceroute during these times shows
the high latency is at the final hop.
I noticed some oddities, here is a full DMESG and my kernel config:
--------------------------------------------------------------------
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p2 #0: Sat Dec 18 02:49:02 EST 2010
alex@svr.mydomain.net:/usr/obj/usr/src/sys/custom amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2398.80-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c
Stepping = 2
Features=0x1781fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,<b31>>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 1572864000 (1500 MB)
avail memory = 1508638720 (1438 MB)
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
kbd1 at kbdmux0
acpi0: <Xen> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x1f48-0x1f4b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, USB> at device 1.2 (no driver attached)
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem
0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
pci0: <unknown> at device 3.0 (no driver attached)
re0: <RealTek 8139C+ 10/100BaseTX> port 0xc100-0xc1ff mem
0xf3001000-0xf30010ff irq 32 at device 4.0 on pci0
re0: Chip rev. 0x74800000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rlphy0: <RealTek internal media interface> PHY 0 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re0: Ethernet address: 00:16:3e:f0:8b:6c
re0: [FILTER]
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROM> at iomem 0xc9000-0xc97ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad0: 46080MB <QEMU HARDDISK 0.10.2> at ata0-master WDMA2
acd0: CDROM <QEMU DVD-ROM/0.10.2> at ata1-master WDMA2
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/ad0s1a
calcru: runtime went backwards from 23875254 usec to 22130573 usec for
pid 1414 (csh)
calcru: runtime went backwards from 5675 usec to 5054 usec for pid 1413 (su)
calcru: runtime went backwards from 19908 usec to 17728 usec for pid
1395 (bash)
calcru: runtime went backwards from 74003 usec to 65901 usec for pid
1395 (bash)
calcru: runtime went backwards from 27707 usec to 24673 usec for pid
1390 (sshd)
calcru: runtime went backwards from 16882 usec to 15033 usec for pid
1390 (sshd)
calcru: runtime went backwards from 2960 usec to 2636 usec for pid 1377
(getty)
calcru: runtime went backwards from 4201 usec to 3741 usec for pid 1376
(getty)
calcru: runtime went backwards from 2334 usec to 2078 usec for pid 1375
(getty)
calcru: runtime went backwards from 3360 usec to 2992 usec for pid 1374
(getty)
calcru: runtime went backwards from 4099 usec to 3650 usec for pid 1373
(getty)
calcru: runtime went backwards from 3052 usec to 2718 usec for pid 1372
(getty)
calcru: runtime went backwards from 3001 usec to 2673 usec for pid 1371
(getty)
calcru: runtime went backwards from 3503 usec to 3120 usec for pid 1370
(getty)
calcru: runtime went backwards from 2561 usec to 2280 usec for pid 1293
(sshd)
calcru: runtime went backwards from 20635 usec to 18376 usec for pid
1171 (sh)
calcru: runtime went backwards from 88696 usec to 78985 usec for pid
1171 (sh)
calcru: runtime went backwards from 231 usec to 205 usec for pid 675
(dhclient)
calcru: runtime went backwards from 2342 usec to 2086 usec for pid 659
(dhclient)
calcru: runtime went backwards from 25186 usec to 22429 usec for pid 659
(dhclient)
calcru: runtime went backwards from 464 usec to 413 usec for pid 110
(adjkerntz)
calcru: runtime went backwards from 10422 usec to 9861 usec for pid 1 (init)
calcru: runtime went backwards from 61599028 usec to 56035925 usec for
pid 1906 (httpd)
calcru: runtime went backwards from 2636 usec to 2063 usec for pid 1377
(getty)
calcru: runtime went backwards from 3741 usec to 2928 usec for pid 1376
(getty)
calcru: runtime went backwards from 2078 usec to 1626 usec for pid 1375
(getty)
calcru: runtime went backwards from 2992 usec to 2341 usec for pid 1374
(getty)
calcru: runtime went backwards from 3650 usec to 2857 usec for pid 1373
(getty)
calcru: runtime went backwards from 2718 usec to 2127 usec for pid 1372
(getty)
calcru: runtime went backwards from 2673 usec to 2092 usec for pid 1371
(getty)
calcru: runtime went backwards from 3120 usec to 2441 usec for pid 1370
(getty)
calcru: runtime went backwards from 2511 usec to 1965 usec for pid 1337
(inetd)
calcru: runtime went backwards from 88691784 usec to 69414995 usec for
pid 1337 (inetd)
calcru: runtime went backwards from 96115 usec to 76173 usec for pid
1293 (sshd)
calcru: runtime went backwards from 18376 usec to 14382 usec for pid
1171 (sh)
calcru: runtime went backwards from 78985 usec to 61818 usec for pid
1171 (sh)
calcru: runtime went backwards from 1542 usec to 1306 usec for pid 676
(devd)
calcru: runtime went backwards from 2086 usec to 1632 usec for pid 659
(dhclient)
calcru: runtime went backwards from 22429 usec to 17554 usec for pid 659
(dhclient)
calcru: runtime went backwards from 413 usec to 323 usec for pid 110
(adjkerntz)
calcru: runtime went backwards from 10170 usec to 8126 usec for pid 1 (init)
calcru: runtime went backwards from 111215702 usec to 89125025 usec for
pid 1 (init)
----------------------------------------------------------------------
[alex@svr ~]$ cat /usr/src/sys/amd64/conf/custom
#
cpu HAMMER
ident custom
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for
devices.
# Use the following to compile in values accessible to the kernel
# through getenv() (or kenv(1) in userland). The format of the file
# is 'variable=value', see kenv(1)
#
# env "GENERIC.env"
makeoptions DEBUG=-g # Build kernel with gdb(1) debug
symbols
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options UFS_GJOURNAL # Enable gjournal-based UFS
journaling
options MD_ROOT # MD is a potential root device
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty)
options COMPAT_FREEBSD32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options P1003_1B_SEMAPHORES # POSIX-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options PRINTF_BUFR_SIZE=128 # Prevent printf output being
interspersed.
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options HWPMC_HOOKS # Necessary kernel hooks for
hwpmc(4)
options AUDIT # Security event auditing
options MAC # TrustedBSD MAC Framework
options FLOWTABLE # per-cpu routing cache
#options KDTRACE_FRAME # Ensure frames are compiled in
#options KDTRACE_HOOKS # Kernel DTrace hooks
options INCLUDE_CONFIG_FILE # Include this file in kernel
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# CPU frequency control
device cpufreq
# Bus support.
device acpi
device pci
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device da # Direct Access (disks)
device cd # CD
device pass # Passthrough device (direct SCSI access)
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device re # RealTek 8139C+/8169/8169S/8110S
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device vlan # 802.1Q VLAN support
device tun # Packet tunnel.
device pty # BSD-style compatibility pseudo ttys
device md # Memory "disks"
device firmware # firmware assist module
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
device pf
device pflog
device pfsync
# Xen HVM support
#options XENHVM
#device xenpci
-------------------------------------------
The XENHVM options commented out causes the VM not to boot (seems to
hang after saying it's found a generic block device attached to the
xenpci bus (virtual hard disk) and that it's adding it as ad0. The
calcru: runtime messages are rather puzzling, perhaps this is related to
the hangs I am seeing regularly. Any ideas on where to start looking?
Thanks!
Alex.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D0F175D.8070701>