Date: Sun, 16 Jun 2013 08:39:42 +0200 From: Andre Albsmeier <Andre.Albsmeier@siemens.com> To: John Baldwin <jhb@freebsd.org> Cc: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: Re: FreeBSD-9.1: machine reboots during snapshot creation, LORs found Message-ID: <20130616063942.GA72803@bali> In-Reply-To: <201305311051.03157.jhb@freebsd.org> References: <20130531122611.GA6607@bali> <201305311051.03157.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 31-May-2013 at 16:51:03 +0200, John Baldwin wrote: > On Friday, May 31, 2013 8:26:11 am Andre Albsmeier wrote: > > Each day at 5:15 we are generating snapshots on various machines. > > This used to work perfectly under 7-STABLE for years but since > > we started to use 9.1-STABLE the machine reboots in about 10% > > of all cases. > > > > After rebooting we find a new snapshot file which is a bit > > smaller than the good ones and with different permissions > > It does not succeed a fsck. In this example it is the one > > whose name is beginning with s3: > > > > -r--r----- 1 root operator snapshot 72802894528 29 May 05:15 s2-2013.05.28-03.15.04 > > -r-------- 1 root operator snapshot 72802893824 29 May 05:15 s3-2013.05.29-03.15.03 > > -r--r----- 1 root operator snapshot 72802894528 28 May 14:22 s4-2013.05.23-06.38.44 > > -r--r----- 1 root operator snapshot 72802894528 28 May 14:22 s5-2013.05.24-03.15.03 > > -r--r----- 1 root operator snapshot 72802894528 28 May 14:22 s6-2013.05.25-03.15.03 > > > > After enabling DIAGNOSTIC, WITNESS and INVARIANTS in the kernel > > I see the following LORs (mksnap_ffs starts exactly at 5:15): > > > > May 29 05:15:00 <kern.crit> palveli kernel: lock order reversal: > > May 29 05:15:00 <kern.crit> palveli kernel: 1st 0xc2371da8 ufs (ufs) @ /src/src-9/sys/kern/vfs_mount.c:1240 > > May 29 05:15:00 <kern.crit> palveli kernel: 2nd 0xc2371ec4 devfs (devfs) @ /src/src-9/sys/ufs/ffs/ffs_vfsops.c:1414 > > May 29 05:15:04 <kern.crit> palveli kernel: lock order reversal: > > May 29 05:15:04 <kern.crit> palveli kernel: 1st 0xc228471c snaplk (snaplk) @ /src/src-9/sys/ufs/ufs/ufs_vnops.c:976 > > May 29 05:15:04 <kern.crit> palveli kernel: 2nd 0xc22f25e4 ufs (ufs) @ /src/src-9/sys/ufs/ffs/ffs_snapshot.c:1626 > > > > Unfortunatley no corefiles are being generated ;-(. > > > > I have checked and even rebuilt the (UFS1) fs in question > > from scratch. I have also seen this happen on an UFS2 on > > another machine and on a third one when running "dump -L" > > on a root fs. > > > > Any hints of how to proceed? > > Would it be possible to setup a serial console that is logged on this machine > to see if it is panic'ing but failing to write out a crashdump? Couldn't attach the serial console yet ;-(. But I had people attach a KVMoverIP switch and enabled the various KDB options in the kernel. Now we can see a bit more (see below) -- no crashdump is being generated though. Some comments on what the crontab script does at 5:01 (I switched it from 5:15 to 5:01 for some reason): 1. Unmount all snapshots 2. Remove all /dev/md devices 3. Deleting the oldest snapshot 4. Generating a new snapshost 5. mdconfig and mount mount of all snapshots I assume the first LOR (sys_unmount) is related to the unmount and the second one (sys_unlink) to the rm. I have added some sleep(1) and sync(1) commands between the different steps but this didn't help. Now the log of three days, we can see another LOR after booting: --- cronjob start, day 1 --- Jun 11 05:01:00 <kern.crit> typhon kernel: lock order reversal: Jun 11 05:01:00 <kern.crit> typhon kernel: 1st 0xc53644c8 ufs (ufs) @ /src/src-9/sys/kern/vfs_mount.c:1240 Jun 11 05:01:00 <kern.crit> typhon kernel: 2nd 0xc5361290 devfs (devfs) @ /src/src-9/sys/ufs/ffs/ffs_vfsops.c:1414 Jun 11 05:01:00 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 11 05:01:00 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,662f7366,765f7366,706f7366,3a632e73,...) at db_trace_self_wrapper+0x26/frame 0xec3c98d4 Jun 11 05:01:00 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198b4,c09e8bb0,586,ec3c99a0,...) at kdb_backtrace+0x2a/frame 0xec3c9930 Jun 11 05:01:00 <kern.crit> typhon kernel: _witness_debugger(c08198b4,c5361290,c080d6e7,c4c29338,c0835390,...) at _witness_debugger+0x25/frame 0xec3c9948 Jun 11 05:01:00 <kern.crit> typhon kernel: witness_checkorder(c5361290,9,c0835390,586,c53612b0,...) at witness_checkorder+0x86f/frame 0xec3c99a0 Jun 11 05:01:00 <kern.crit> typhon kernel: __lockmgr_args(c5361290,80400,c53612b0,0,0,...) at __lockmgr_args+0x829/frame 0xec3c9a58 Jun 11 05:01:00 <kern.crit> typhon kernel: vop_stdlock(ec3c9abc,246,c08bcd9c,80400,c5361238,...) at vop_stdlock+0x62/frame 0xec3c9a8c Jun 11 05:01:00 <kern.crit> typhon kernel: VOP_LOCK1_APV(c08611e0,ec3c9abc,c09e8bb4,c0890120,c5361238,...) at VOP_LOCK1_APV+0xb5/frame 0xec3c9aa8 Jun 11 05:01:00 <kern.crit> typhon kernel: _vn_lock(c5361238,80400,c0835390,586,ec3c9b14,...) at _vn_lock+0x5e/frame 0xec3c9adc Jun 11 05:01:00 <kern.crit> typhon kernel: ffs_flushfiles(c5365d34,0,c67ec600,0,c5365d34,...) at ffs_flushfiles+0x133/frame 0xec3c9b1c Jun 11 05:01:00 <kern.crit> typhon kernel: ffs_unmount(c5365d34,8000000,c0821043,513,c4c00c08,...) at ffs_unmount+0x180/frame 0xec3c9b5c Jun 11 05:01:00 <kern.crit> typhon kernel: dounmount(c5365d34,8000000,c67ec600,494,c67e8378,...) at dounmount+0x423/frame 0xec3c9bac Jun 11 05:01:00 <kern.crit> typhon kernel: sys_unmount(c67ec600,ec3c9ccc,c0846650,c081a478,206,...) at sys_unmount+0x3d1/frame 0xec3c9c48 Jun 11 05:01:00 <kern.crit> typhon kernel: syscall(ec3c9d08) at syscall+0x24d/frame 0xec3c9cfc Jun 11 05:01:00 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xec3c9cfc Jun 11 05:01:00 <kern.crit> typhon kernel: --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x180bf3e7, esp = 0xbfbfd61c, ebp = 0xbfbfd6e8 --- Jun 11 05:01:10 <kern.crit> typhon kernel: lock order reversal: Jun 11 05:01:10 <kern.crit> typhon kernel: 1st 0xc503545c snaplk (snaplk) @ /src/src-9/sys/ufs/ufs/ufs_vnops.c:976 Jun 11 05:01:10 <kern.crit> typhon kernel: 2nd 0xc509eda8 ufs (ufs) @ /src/src-9/sys/ufs/ffs/ffs_snapshot.c:1626 Jun 11 05:01:10 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 11 05:01:10 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,735f7366,7370616e,2e746f68,36313a63,...) at db_trace_self_wrapper+0x26/frame 0xec3bd614 Jun 11 05:01:10 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198b4,c09e96f8,65a,ec3bd6e0,...) at kdb_backtrace+0x2a/frame 0xec3bd670 Jun 11 05:01:10 <kern.crit> typhon kernel: _witness_debugger(c08198b4,c509eda8,c0808f48,c4c29470,c0831f9d,...) at _witness_debugger+0x25/frame 0xec3bd688 Jun 11 05:01:10 <kern.crit> typhon kernel: witness_checkorder(c509eda8,9,c0831f9d,65a,0,...) at witness_checkorder+0x86f/frame 0xec3bd6e0 Jun 11 05:01:10 <kern.crit> typhon kernel: __lockmgr_args(c509eda8,80000,0,0,0,...) at __lockmgr_args+0x829/frame 0xec3bd798 Jun 11 05:01:10 <kern.crit> typhon kernel: ffs_snapremove(c509ed50,c4c23f88,c4c26140,4,c09e96f8,...) at ffs_snapremove+0x11f/frame 0xec3bd818 Jun 11 05:01:10 <kern.crit> typhon kernel: ffs_truncate(c509ed50,0,0,c00,0,...) at ffs_truncate+0x637/frame 0xec3bda44 Jun 11 05:01:10 <kern.crit> typhon kernel: ufs_inactive(ec3bdab8,c509ed50,c509ed50,c535d1bc,ec3bdad0,...) at ufs_inactive+0x1f8/frame 0xec3bda80 Jun 11 05:01:10 <kern.crit> typhon kernel: VOP_INACTIVE_APV(c0882560,ec3bdab8,c08219ca,a00,c068657a,...) at VOP_INACTIVE_APV+0xa5/frame 0xec3bda98 Jun 11 05:01:10 <kern.crit> typhon kernel: vinactive(c509ed50,c67ee300,c08219ca,8fe,0,...) at vinactive+0x112/frame 0xec3bdad0 Jun 11 05:01:10 <kern.crit> typhon kernel: vputx(ec3bdc18,c069fcf4,c509ed50,ffffffdf,2,...) at vputx+0x2fc/frame 0xec3bdb0c Jun 11 05:01:10 <kern.crit> typhon kernel: vput(c509ed50,ffffffdf,2,0,5,...) at vput+0x10/frame 0xec3bdb14 Jun 11 05:01:10 <kern.crit> typhon kernel: kern_unlinkat(c67ee300,ffffff9c,bfbfdf32,0,0,...) at kern_unlinkat+0x274/frame 0xec3bdc18 Jun 11 05:01:10 <kern.crit> typhon kernel: kern_unlink(c67ee300,bfbfdf32,0,ec3bdcfc,c07b310d,...) at kern_unlink+0x2f/frame 0xec3bdc34 Jun 11 05:01:10 <kern.crit> typhon kernel: sys_unlink(c67ee300,ec3bdccc,c0846650,c081a4d8,246,...) at sys_unlink+0x22/frame 0xec3bdc48 Jun 11 05:01:10 <kern.crit> typhon kernel: syscall(ec3bdd08) at syscall+0x24d/frame 0xec3bdcfc Jun 11 05:01:10 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xec3bdcfc Jun 11 05:01:10 <kern.crit> typhon kernel: --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x1814ee83, esp = 0xbfbfdd3c, ebp = 0xbfbfddb8 --- --- reboot on day 1 --- Jun 11 05:35:46 <kern.crit> typhon kernel: Copyright (c) 1992-2013 The FreeBSD Project. Jun 11 05:35:46 <kern.crit> typhon kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jun 11 05:35:46 <kern.crit> typhon kernel: The Regents of the University of California. All rights reserved. Jun 11 05:35:46 <kern.crit> typhon kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Jun 11 05:35:46 <kern.crit> typhon kernel: FreeBSD 9.1-STABLE #3: Sat Jun 8 11:02:16 CEST 2013 Jun 11 05:35:46 <kern.crit> typhon kernel: root@server.ofw.tld:/usr/obj/src/src-9/sys/typhon i386 Jun 11 05:35:46 <kern.crit> typhon kernel: gcc version 4.2.1 20070831 patched [FreeBSD] Jun 11 05:35:46 <kern.crit> typhon kernel: WARNING: WITNESS option enabled, expect reduced performance. Jun 11 05:35:46 <kern.crit> typhon kernel: WARNING: DIAGNOSTIC option enabled, expect reduced performance. Jun 11 05:35:46 <kern.crit> typhon kernel: CPU: Pentium(R) Dual-Core CPU E6600 @ 3.06GHz (3071.98-MHz 686-class CPU) Jun 11 05:35:46 <kern.crit> typhon kernel: Origin = "GenuineIntel" Id = 0x1067a Family = 0x6 Model = 0x17 Stepping = 10 Jun 11 05:35:46 <kern.crit> typhon kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Jun 11 05:35:46 <kern.crit> typhon kernel: Features2=0x400e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE> Jun 11 05:35:46 <kern.crit> typhon kernel: AMD Features=0x20100000<NX,LM> Jun 11 05:35:46 <kern.crit> typhon kernel: AMD Features2=0x1<LAHF> Jun 11 05:35:46 <kern.crit> typhon kernel: TSC: P-state invariant, performance statistics Jun 11 05:35:46 <kern.crit> typhon kernel: real memory = 2147483648 (2048 MB) Jun 11 05:35:46 <kern.crit> typhon kernel: avail memory = 2096226304 (1999 MB) Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "LAPIC" quality 400 Jun 11 05:35:46 <kern.crit> typhon kernel: ACPI APIC Table: <A_M_I_ OEMAPIC > Jun 11 05:35:46 <kern.crit> typhon kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs Jun 11 05:35:46 <kern.crit> typhon kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) Jun 11 05:35:46 <kern.crit> typhon kernel: cpu0 (BSP): APIC ID: 0 Jun 11 05:35:46 <kern.crit> typhon kernel: cpu1 (AP): APIC ID: 1 Jun 11 05:35:46 <kern.crit> typhon kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard Jun 11 05:35:46 <kern.crit> typhon kernel: acpi0: <A_M_I_ OEMXSDT> on motherboard Jun 11 05:35:46 <kern.crit> typhon kernel: acpi0: Power Button (fixed) Jun 11 05:35:46 <kern.crit> typhon kernel: acpi0: reservation of 0, a0000 (3) failed Jun 11 05:35:46 <kern.crit> typhon kernel: acpi0: reservation of 100000, 7ff00000 (3) failed Jun 11 05:35:46 <kern.crit> typhon kernel: cpu0: <ACPI CPU> on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: cpu1: <ACPI CPU> on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "i8254" frequency 1193182 Hz quality 100 Jun 11 05:35:46 <kern.crit> typhon kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "RTC" frequency 32768 Hz quality 0 Jun 11 05:35:46 <kern.crit> typhon kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: Timecounter "HPET" frequency 14318180 Hz quality 950 Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "HPET" frequency 14318180 Hz quality 450 Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "HPET1" frequency 14318180 Hz quality 440 Jun 11 05:35:46 <kern.crit> typhon kernel: Event timer "HPET2" frequency 14318180 Hz quality 440 Jun 11 05:35:46 <kern.crit> typhon kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 Jun 11 05:35:46 <kern.crit> typhon kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: pci0: <ACPI PCI bus> on pcib0 Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: RAM ECC Monitor v0.13 on i975X (8086:277C), reporting tested Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: Capabilities: ECC detection and correction Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: Current mode: ECC detection and correction Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: Bank Size Type ILV ECC Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: 0 1024M DDR2 Y Y Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: 1 1024M DDR2 Y Y Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: Total RAM detected: 2048M Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: <RAM ECC Monitor v0.13> on hostb0 Jun 11 05:35:46 <kern.crit> typhon kernel: eccmon0: attached Jun 11 05:35:46 <kern.crit> typhon kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: pci4: <ACPI PCI bus> on pcib1 Jun 11 05:35:46 <kern.crit> typhon kernel: 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 Jun 11 05:35:46 <kern.crit> typhon kernel: twa0: <3ware 9000 series Storage Controller> port 0xd800-0xd8ff mem 0xfc000000-0xfdffffff,0xff9ff000-0xff9fffff irq 16 at device 0.0 on pci4 Jun 11 05:35:46 <kern.crit> typhon kernel: twa0: INFO: (0x04: 0x0001): Controller reset occurred: resets=1 Jun 11 05:35:46 <kern.crit> typhon kernel: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9690SA-8I, 128 ports, Firmware FH9X 4.10.00.027, BIOS BE9X 4.08.00.004 Jun 11 05:35:46 <kern.crit> typhon kernel: pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: pci3: <ACPI PCI bus> on pcib2 Jun 11 05:35:46 <kern.crit> typhon kernel: pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: pci2: <ACPI PCI bus> on pcib3 Jun 11 05:35:46 <kern.crit> typhon kernel: mskc0: <Marvell Yukon 88E8053 Gigabit Ethernet> port 0xc800-0xc8ff mem 0xff8fc000-0xff8fffff irq 19 at device 0.0 on pci2 Jun 11 05:35:46 <kern.crit> typhon kernel: msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0 Jun 11 05:35:46 <kern.crit> typhon kernel: msk0: Ethernet address: 00:18:f3:87:86:60 Jun 11 05:35:46 <kern.crit> typhon kernel: miibus0: <MII bus> on msk0 Jun 11 05:35:46 <kern.crit> typhon kernel: e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0 Jun 11 05:35:46 <kern.crit> typhon kernel: e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow Jun 11 05:35:46 <kern.crit> typhon kernel: pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: pci1: <ACPI PCI bus> on pcib4 Jun 11 05:35:46 <kern.crit> typhon kernel: vgapci0: <VGA-compatible display> mem 0xff7fc000-0xff7fffff,0xfa000000-0xfa7fffff irq 21 at device 0.0 on pci1 Jun 11 05:35:46 <kern.crit> typhon kernel: ahc0: <Adaptec 29160 Ultra160 SCSI adapter> port 0xb800-0xb8ff mem 0xff7fb000-0xff7fbfff irq 22 at device 1.0 on pci1 Jun 11 05:35:46 <kern.crit> typhon kernel: ahc0: Bugs (0x0040): SCBCHAN_UPLOAD Jun 11 05:35:46 <kern.crit> typhon kernel: aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs Jun 11 05:35:46 <kern.crit> typhon kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: isa0: <ISA bus> on isab0 Jun 11 05:35:46 <kern.crit> typhon kernel: atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: ata0: <ATA channel> at channel 0 on atapci0 Jun 11 05:35:46 <kern.crit> typhon kernel: atapci1: <Intel ICH7 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f mem 0xffaffc00-0xffafffff irq 23 at device 31.2 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: ata2: <ATA channel> at channel 0 on atapci1 Jun 11 05:35:46 <kern.crit> typhon kernel: ata3: <ATA channel> at channel 1 on atapci1 Jun 11 05:35:46 <kern.crit> typhon kernel: ichsmb0: <Intel 82801GB (ICH7) SMBus controller> port 0x400-0x41f at device 31.3 on pci0 Jun 11 05:35:46 <kern.crit> typhon kernel: smbus0: <System Management Bus> on ichsmb0 Jun 11 05:35:46 <kern.crit> typhon kernel: smb0: <SMBus generic I/O> on smbus0 Jun 11 05:35:46 <kern.crit> typhon kernel: acpi_button0: <Power Button> on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0 Jun 11 05:35:46 <kern.crit> typhon kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0 Jun 11 05:35:46 <kern.crit> typhon kernel: atkbd0: [GIANT-LOCKED] Jun 11 05:35:46 <kern.crit> typhon kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 Jun 11 05:35:46 <kern.crit> typhon kernel: ichwd0 on isa0 Jun 11 05:35:46 <kern.crit> typhon kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcd800-0xcdfff pnpid ORM0000 on isa0 Jun 11 05:35:46 <kern.crit> typhon kernel: sc0: <System console> at flags 0x100 on isa0 Jun 11 05:35:46 <kern.crit> typhon kernel: sc0: VGA <9 virtual consoles, flags=0x300> Jun 11 05:35:46 <kern.crit> typhon kernel: vga0: <Generic ISA VGA> at port 0x3b0-0x3df iomem 0xa0000-0xbffff on isa0 Jun 11 05:35:46 <kern.crit> typhon kernel: coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Jun 11 05:35:46 <kern.crit> typhon kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0 Jun 11 05:35:46 <kern.crit> typhon kernel: est0: Guessed bus clock (high) of 267 MHz Jun 11 05:35:46 <kern.crit> typhon kernel: est0: Guessed high setting of 3071 MHz @ 1292 mV Jun 11 05:35:46 <kern.crit> typhon kernel: est0: Guessed low setting of 1602 MHz @ 1052 mV Jun 11 05:35:46 <kern.crit> typhon kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0 Jun 11 05:35:46 <kern.crit> typhon kernel: coretemp1: <CPU On-Die Thermal Sensors> on cpu1 Jun 11 05:35:46 <kern.crit> typhon kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1 Jun 11 05:35:46 <kern.crit> typhon kernel: est1: Guessed bus clock (high) of 267 MHz Jun 11 05:35:46 <kern.crit> typhon kernel: est1: Guessed high setting of 3071 MHz @ 1292 mV Jun 11 05:35:46 <kern.crit> typhon kernel: est1: Guessed low setting of 1602 MHz @ 1052 mV Jun 11 05:35:46 <kern.crit> typhon kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1 Jun 11 05:35:46 <kern.crit> typhon kernel: Timecounters tick every 10.000 msec Jun 11 05:35:46 <kern.crit> typhon kernel: Expensive timeout(9) function: 0xc04d8900(0xc4d8e800) 0.123587844 s Jun 11 05:35:46 <kern.crit> typhon kernel: da10 at twa0 bus 0 scbus1 target 0 lun 0 Jun 11 05:35:46 <kern.crit> typhon kernel: da10: <AMCC 9690SA-8I DISK 4.10> Fixed Direct Access SCSI-5 device Jun 11 05:35:46 <kern.crit> typhon kernel: da10: 100.000MB/s transfers Jun 11 05:35:46 <kern.crit> typhon kernel: da10: 2860962MB (5859250176 512 byte sectors: 255H 63S/T 364721C) Jun 11 05:35:46 <kern.crit> typhon kernel: SMP: AP CPU #1 Launched! Jun 11 05:35:46 <kern.crit> typhon kernel: Timecounter "TSC-low" frequency 1535990772 Hz quality 1000 Jun 11 05:35:46 <kern.crit> typhon kernel: WARNING: WITNESS option enabled, expect reduced performance. Jun 11 05:35:46 <kern.crit> typhon kernel: WARNING: DIAGNOSTIC option enabled, expect reduced performance. Jun 11 05:35:46 <kern.crit> typhon kernel: da1 at ahc0 bus 0 scbus0 target 1 lun 0 Jun 11 05:35:46 <kern.crit> typhon kernel: da1: <IBM DDRS-39130W S97B> Fixed Direct Access SCSI-2 device Jun 11 05:35:46 <kern.crit> typhon kernel: da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) Jun 11 05:35:46 <kern.crit> typhon kernel: da1: Command Queueing enabled Jun 11 05:35:46 <kern.crit> typhon kernel: da1: 8715MB (17850000 512 byte sectors: 255H 63S/T 1111C) Jun 11 05:35:46 <kern.crit> typhon kernel: da0 at ahc0 bus 0 scbus0 target 0 lun 0 Jun 11 05:35:46 <kern.crit> typhon kernel: da0: <IBM DDRS-39130W S92A> Fixed Direct Access SCSI-2 device Jun 11 05:35:46 <kern.crit> typhon kernel: da0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) Jun 11 05:35:46 <kern.crit> typhon kernel: da0: Command Queueing enabled Jun 11 05:35:46 <kern.crit> typhon kernel: da0: 8715MB (17850000 512 byte sectors: 255H 63S/T 1111C) Jun 11 05:35:46 <kern.crit> typhon kernel: Trying to mount root from ufs:/dev/da0s1a [noatime,rw]... Jun 11 05:35:46 <kern.crit> typhon kernel: WARNING: / was not properly dismounted Jun 11 05:38:10 <kern.crit> typhon kernel: done. --- LOR after reboot --- Jun 11 05:38:12 <kern.crit> typhon kernel: lock order reversal: Jun 11 05:38:12 <kern.crit> typhon kernel: 1st 0xde887868 bufwait (bufwait) @ /src/src-9/sys/kern/vfs_bio.c:2725 Jun 11 05:38:12 <kern.crit> typhon kernel: 2nd 0xc503559c snaplk (snaplk) @ /src/src-9/sys/ufs/ffs/ffs_snapshot.c:2296 Jun 11 05:38:12 <kern.crit> typhon kernel: 3rd 0xde887b10 bufwait (bufwait) @ /src/src-9/sys/kern/vfs_bio.c:2725 Jun 11 05:38:12 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 11 05:38:12 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,2f6e7265,5f736676,2e6f6962,37323a63,...) at db_trace_self_wrapper+0x26/frame 0xdebf63e0 Jun 11 05:38:12 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198cd,c09e9990,aa5,debf64ac,...) at kdb_backtrace+0x2a/frame 0xdebf643c Jun 11 05:38:12 <kern.crit> typhon kernel: _witness_debugger(c08198cd,de887b10,c081fa43,c4c26140,c081ee1e,...) at _witness_debugger+0x25/frame 0xdebf6454 Jun 11 05:38:12 <kern.crit> typhon kernel: witness_checkorder(de887b10,9,c081ee1e,aa5,0,...) at witness_checkorder+0x86f/frame 0xdebf64ac Jun 11 05:38:12 <kern.crit> typhon kernel: __lockmgr_args(de887b10,81900,c509df18,c081f99f,60,...) at __lockmgr_args+0x829/frame 0xdebf6564 Jun 11 05:38:12 <kern.crit> typhon kernel: getblk(c509de6c,fffff7f3,ffffffff,4000,0,...) at getblk+0x154/frame 0xdebf65b8 Jun 11 05:38:12 <kern.crit> typhon kernel: breadn_flags(c509de6c,fffff7f3,ffffffff,4000,0,...) at breadn_flags+0x43/frame 0xdebf65e8 Jun 11 05:38:12 <kern.crit> typhon kernel: bread(c509de6c,fffff7f3,ffffffff,4000,0,...) at bread+0x54/frame 0xdebf6618 Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_balloc_ufs2(c509de6c,918f8000,1,4000,c4c0fb00,...) at ffs_balloc_ufs2+0x16ee/frame 0xdebf67c4 Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_copyonwrite(c509d8e0,de887808,de887808,0,0,...) at ffs_copyonwrite+0x3ee/frame 0xdebf6840 Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_geom_strategy(c509d98c,de887808,0,de887808,3231b8,...) at ffs_geom_strategy+0xeb/frame 0xdebf685c Jun 11 05:38:12 <kern.crit> typhon kernel: bufwrite(de887808,0,c0835390,832,0) at bufwrite+0x169/frame 0xdebf687c Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_bufwrite(de887808,c5119300,100,4000,0,...) at ffs_bufwrite+0x290/frame 0xdebf68a0 Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_update(c53aa238,1,c0835afd,152,0,...) at ffs_update+0x395/frame 0xdebf6910 Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_syncvnode(c53aa238,1,0,4c6,c53aa2e4,...) at ffs_syncvnode+0x591/frame 0xdebf696c Jun 11 05:38:12 <kern.crit> typhon kernel: ffs_fsync(debf69bc,4ce,c08219ca,0,debf69cc,...) at ffs_fsync+0x2f/frame 0xdebf6994 Jun 11 05:38:12 <kern.crit> typhon kernel: VOP_FSYNC_APV(c0882560,debf69bc,c088ffc0,c53aa238,1,...) at VOP_FSYNC_APV+0xa5/frame 0xdebf69ac Jun 11 05:38:12 <kern.crit> typhon kernel: bufsync(c53aa2e4,1,c08219ca,4ce,c4fb3d00,...) at bufsync+0x38/frame 0xdebf69cc Jun 11 05:38:12 <kern.crit> typhon kernel: bufobj_invalbuf(c53aa2e4,1,0,0,debf6c24,...) at bufobj_invalbuf+0xb5/frame 0xdebf6a04 Jun 11 05:38:12 <kern.crit> typhon kernel: vinvalbuf(c53aa238,1,0,0,0,...) at vinvalbuf+0x2b/frame 0xdebf6a1c Jun 11 05:38:12 <kern.crit> typhon kernel: vfs_donmount(c5023600,5,0,c52e3300,c52e3300,...) at vfs_donmount+0xabe/frame 0xdebf6c24 Jun 11 05:38:12 <kern.crit> typhon kernel: sys_nmount(c5023600,debf6ccc,c0846650,c081a5d6,286,...) at sys_nmount+0x65/frame 0xdebf6c48 Jun 11 05:38:12 <kern.crit> typhon kernel: syscall(debf6d08) at syscall+0x24d/frame 0xdebf6cfc Jun 11 05:38:12 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xdebf6cfc Jun 11 05:38:12 <kern.crit> typhon kernel: --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x180d20c7, esp = 0xbfbfcdbc, ebp = 0xbfbfd318 --- --- cronjob start, day 2 --- Jun 12 05:01:00 <kern.crit> typhon kernel: lock order reversal: Jun 12 05:01:00 <kern.crit> typhon kernel: 1st 0xc515d290 ufs (ufs) @ /src/src-9/sys/kern/vfs_mount.c:1240 Jun 12 05:01:00 <kern.crit> typhon kernel: 2nd 0xc53aa058 devfs (devfs) @ /src/src-9/sys/ufs/ffs/ffs_vfsops.c:1414 Jun 12 05:01:00 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 12 05:01:00 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,662f7366,765f7366,706f7366,3a632e73,...) at db_trace_self_wrapper+0x26/frame 0xec36e8d4 Jun 12 05:01:00 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198b4,c09e9a20,586,ec36e9a0,...) at kdb_backtrace+0x2a/frame 0xec36e930 Jun 12 05:01:00 <kern.crit> typhon kernel: _witness_debugger(c08198b4,c53aa058,c080d6e7,c4c29338,c0835390,...) at _witness_debugger+0x25/frame 0xec36e948 Jun 12 05:01:00 <kern.crit> typhon kernel: witness_checkorder(c53aa058,9,c0835390,586,c53aa078,...) at witness_checkorder+0x86f/frame 0xec36e9a0 Jun 12 05:01:00 <kern.crit> typhon kernel: __lockmgr_args(c53aa058,80400,c53aa078,0,0,...) at __lockmgr_args+0x829/frame 0xec36ea58 Jun 12 05:01:00 <kern.crit> typhon kernel: vop_stdlock(ec36eabc,246,c08bcd9c,80400,c53aa000,...) at vop_stdlock+0x62/frame 0xec36ea8c Jun 12 05:01:00 <kern.crit> typhon kernel: VOP_LOCK1_APV(c08611e0,ec36eabc,c09e9a24,c0890120,c53aa000,...) at VOP_LOCK1_APV+0xb5/frame 0xec36eaa8 Jun 12 05:01:00 <kern.crit> typhon kernel: _vn_lock(c53aa000,80400,c0835390,586,ec36eb14,...) at _vn_lock+0x5e/frame 0xec36eadc Jun 12 05:01:00 <kern.crit> typhon kernel: ffs_flushfiles(c504b000,0,c5414000,0,c504b000,...) at ffs_flushfiles+0x133/frame 0xec36eb1c Jun 12 05:01:00 <kern.crit> typhon kernel: ffs_unmount(c504b000,8000000,c0821043,513,c4c00c08,...) at ffs_unmount+0x180/frame 0xec36eb5c Jun 12 05:01:00 <kern.crit> typhon kernel: dounmount(c504b000,8000000,c5414000,494,c519e958,...) at dounmount+0x423/frame 0xec36ebac Jun 12 05:01:00 <kern.crit> typhon kernel: sys_unmount(c5414000,ec36eccc,c0846650,c081a478,206,...) at sys_unmount+0x3d1/frame 0xec36ec48 Jun 12 05:01:00 <kern.crit> typhon kernel: syscall(ec36ed08) at syscall+0x24d/frame 0xec36ecfc Jun 12 05:01:00 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xec36ecfc Jun 12 05:01:00 <kern.crit> typhon kernel: --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x180bf3e7, esp = 0xbfbfd61c, ebp = 0xbfbfd6e8 --- Jun 12 05:01:08 <kern.crit> typhon kernel: lock order reversal: Jun 12 05:01:08 <kern.crit> typhon kernel: 1st 0xc503559c snaplk (snaplk) @ /src/src-9/sys/ufs/ufs/ufs_vnops.c:976 Jun 12 05:01:08 <kern.crit> typhon kernel: 2nd 0xc511b058 ufs (ufs) @ /src/src-9/sys/ufs/ffs/ffs_snapshot.c:1626 Jun 12 05:01:08 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 12 05:01:08 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,735f7366,7370616e,2e746f68,36313a63,...) at db_trace_self_wrapper+0x26/frame 0xec350614 Jun 12 05:01:08 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198b4,c09e9750,65a,ec3506e0,...) at kdb_backtrace+0x2a/frame 0xec350670 Jun 12 05:01:08 <kern.crit> typhon kernel: _witness_debugger(c08198b4,c511b058,c0808f48,c4c29470,c0831f9d,...) at _witness_debugger+0x25/frame 0xec350688 Jun 12 05:01:08 <kern.crit> typhon kernel: witness_checkorder(c511b058,9,c0831f9d,65a,0,...) at witness_checkorder+0x86f/frame 0xec3506e0 Jun 12 05:01:08 <kern.crit> typhon kernel: __lockmgr_args(c511b058,80000,0,0,0,...) at __lockmgr_args+0x829/frame 0xec350798 Jun 12 05:01:08 <kern.crit> typhon kernel: ffs_snapremove(c511b000,c4c23f88,c4c26140,4,c09e9750,...) at ffs_snapremove+0x11f/frame 0xec350818 Jun 12 05:01:08 <kern.crit> typhon kernel: ffs_truncate(c511b000,0,0,c00,0,...) at ffs_truncate+0x637/frame 0xec350a44 Jun 12 05:01:08 <kern.crit> typhon kernel: ufs_inactive(ec350ab8,c511b000,c511b000,c52fcd4c,ec350ad0,...) at ufs_inactive+0x1f8/frame 0xec350a80 Jun 12 05:01:08 <kern.crit> typhon kernel: VOP_INACTIVE_APV(c0882560,ec350ab8,c08219ca,a00,c068657a,...) at VOP_INACTIVE_APV+0xa5/frame 0xec350a98 Jun 12 05:01:08 <kern.crit> typhon kernel: vinactive(c511b000,c51a3c00,c08219ca,8fe,0,...) at vinactive+0x112/frame 0xec350ad0 Jun 12 05:01:08 <kern.crit> typhon kernel: vputx(ec350c18,c069fcf4,c511b000,ffffffdf,2,...) at vputx+0x2fc/frame 0xec350b0c Jun 12 05:01:08 <kern.crit> typhon kernel: vput(c511b000,ffffffdf,2,0,5,...) at vput+0x10/frame 0xec350b14 Jun 12 05:01:08 <kern.crit> typhon kernel: kern_unlinkat(c51a3c00,ffffff9c,bfbfdf32,0,0,...) at kern_unlinkat+0x274/frame 0xec350c18 Jun 12 05:01:08 <kern.crit> typhon kernel: kern_unlink(c51a3c00,bfbfdf32,0,ec350cfc,c07b310d,...) at kern_unlink+0x2f/frame 0xec350c34 Jun 12 05:01:08 <kern.crit> typhon kernel: sys_unlink(c51a3c00,ec350ccc,c0846650,c081a4d8,246,...) at sys_unlink+0x22/frame 0xec350c48 Jun 12 05:01:08 <kern.crit> typhon kernel: syscall(ec350d08) at syscall+0x24d/frame 0xec350cfc Jun 12 05:01:08 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xec350cfc Jun 12 05:01:08 <kern.crit> typhon kernel: --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x1814ee83, esp = 0xbfbfdd3c, ebp = 0xbfbfddb8 --- --- reboot on day 2 --- Jun 12 05:35:46 <kern.crit> typhon kernel: Copyright (c) 1992-2013 The FreeBSD Project. Jun 12 05:35:46 <kern.crit> typhon kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jun 12 05:35:46 <kern.crit> typhon kernel: The Regents of the University of California. All rights reserved. Jun 12 05:35:46 <kern.crit> typhon kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Jun 12 05:35:46 <kern.crit> typhon kernel: FreeBSD 9.1-STABLE #3: Sat Jun 8 11:02:16 CEST 2013 Jun 12 05:35:46 <kern.crit> typhon kernel: root@server.ofw.tld:/usr/obj/src/src-9/sys/typhon i386 Jun 12 05:35:46 <kern.crit> typhon kernel: gcc version 4.2.1 20070831 patched [FreeBSD] Jun 12 05:35:46 <kern.crit> typhon kernel: WARNING: WITNESS option enabled, expect reduced performance. Jun 12 05:35:46 <kern.crit> typhon kernel: WARNING: DIAGNOSTIC option enabled, expect reduced performance. Jun 12 05:35:46 <kern.crit> typhon kernel: CPU: Pentium(R) Dual-Core CPU E6600 @ 3.06GHz (3071.99-MHz 686-class CPU) Jun 12 05:35:46 <kern.crit> typhon kernel: Origin = "GenuineIntel" Id = 0x1067a Family = 0x6 Model = 0x17 Stepping = 10 Jun 12 05:35:46 <kern.crit> typhon kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Jun 12 05:35:46 <kern.crit> typhon kernel: Features2=0x400e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE> Jun 12 05:35:46 <kern.crit> typhon kernel: AMD Features=0x20100000<NX,LM> Jun 12 05:35:46 <kern.crit> typhon kernel: AMD Features2=0x1<LAHF> Jun 12 05:35:46 <kern.crit> typhon kernel: TSC: P-state invariant, performance statistics Jun 12 05:35:46 <kern.crit> typhon kernel: real memory = 2147483648 (2048 MB) Jun 12 05:35:46 <kern.crit> typhon kernel: avail memory = 2096226304 (1999 MB) Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "LAPIC" quality 400 Jun 12 05:35:46 <kern.crit> typhon kernel: ACPI APIC Table: <A_M_I_ OEMAPIC > Jun 12 05:35:46 <kern.crit> typhon kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs Jun 12 05:35:46 <kern.crit> typhon kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) Jun 12 05:35:46 <kern.crit> typhon kernel: cpu0 (BSP): APIC ID: 0 Jun 12 05:35:46 <kern.crit> typhon kernel: cpu1 (AP): APIC ID: 1 Jun 12 05:35:46 <kern.crit> typhon kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard Jun 12 05:35:46 <kern.crit> typhon kernel: acpi0: <A_M_I_ OEMXSDT> on motherboard Jun 12 05:35:46 <kern.crit> typhon kernel: acpi0: Power Button (fixed) Jun 12 05:35:46 <kern.crit> typhon kernel: acpi0: reservation of 0, a0000 (3) failed Jun 12 05:35:46 <kern.crit> typhon kernel: acpi0: reservation of 100000, 7ff00000 (3) failed Jun 12 05:35:46 <kern.crit> typhon kernel: cpu0: <ACPI CPU> on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: cpu1: <ACPI CPU> on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "i8254" frequency 1193182 Hz quality 100 Jun 12 05:35:46 <kern.crit> typhon kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "RTC" frequency 32768 Hz quality 0 Jun 12 05:35:46 <kern.crit> typhon kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: Timecounter "HPET" frequency 14318180 Hz quality 950 Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "HPET" frequency 14318180 Hz quality 450 Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "HPET1" frequency 14318180 Hz quality 440 Jun 12 05:35:46 <kern.crit> typhon kernel: Event timer "HPET2" frequency 14318180 Hz quality 440 Jun 12 05:35:46 <kern.crit> typhon kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 Jun 12 05:35:46 <kern.crit> typhon kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: pci0: <ACPI PCI bus> on pcib0 Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: RAM ECC Monitor v0.13 on i975X (8086:277C), reporting tested Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: Capabilities: ECC detection and correction Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: Current mode: ECC detection and correction Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: Bank Size Type ILV ECC Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: 0 1024M DDR2 Y Y Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: 1 1024M DDR2 Y Y Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: Total RAM detected: 2048M Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: <RAM ECC Monitor v0.13> on hostb0 Jun 12 05:35:46 <kern.crit> typhon kernel: eccmon0: attached Jun 12 05:35:46 <kern.crit> typhon kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: pci4: <ACPI PCI bus> on pcib1 Jun 12 05:35:46 <kern.crit> typhon kernel: 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 Jun 12 05:35:46 <kern.crit> typhon kernel: twa0: <3ware 9000 series Storage Controller> port 0xd800-0xd8ff mem 0xfc000000-0xfdffffff,0xff9ff000-0xff9fffff irq 16 at device 0.0 on pci4 Jun 12 05:35:46 <kern.crit> typhon kernel: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9690SA-8I, 128 ports, Firmware FH9X 4.10.00.027, BIOS BE9X 4.08.00.004 Jun 12 05:35:46 <kern.crit> typhon kernel: pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: pci3: <ACPI PCI bus> on pcib2 Jun 12 05:35:46 <kern.crit> typhon kernel: pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: pci2: <ACPI PCI bus> on pcib3 Jun 12 05:35:46 <kern.crit> typhon kernel: mskc0: <Marvell Yukon 88E8053 Gigabit Ethernet> port 0xc800-0xc8ff mem 0xff8fc000-0xff8fffff irq 19 at device 0.0 on pci2 Jun 12 05:35:46 <kern.crit> typhon kernel: msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0 Jun 12 05:35:46 <kern.crit> typhon kernel: msk0: Ethernet address: 00:18:f3:87:86:60 Jun 12 05:35:46 <kern.crit> typhon kernel: miibus0: <MII bus> on msk0 Jun 12 05:35:46 <kern.crit> typhon kernel: e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0 Jun 12 05:35:46 <kern.crit> typhon kernel: e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow Jun 12 05:35:46 <kern.crit> typhon kernel: pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: pci1: <ACPI PCI bus> on pcib4 Jun 12 05:35:46 <kern.crit> typhon kernel: vgapci0: <VGA-compatible display> mem 0xff7fc000-0xff7fffff,0xfa000000-0xfa7fffff irq 21 at device 0.0 on pci1 Jun 12 05:35:46 <kern.crit> typhon kernel: ahc0: <Adaptec 29160 Ultra160 SCSI adapter> port 0xb800-0xb8ff mem 0xff7fb000-0xff7fbfff irq 22 at device 1.0 on pci1 Jun 12 05:35:46 <kern.crit> typhon kernel: ahc0: Bugs (0x0040): SCBCHAN_UPLOAD Jun 12 05:35:46 <kern.crit> typhon kernel: aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs Jun 12 05:35:46 <kern.crit> typhon kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: isa0: <ISA bus> on isab0 Jun 12 05:35:46 <kern.crit> typhon kernel: atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: ata0: <ATA channel> at channel 0 on atapci0 Jun 12 05:35:46 <kern.crit> typhon kernel: atapci1: <Intel ICH7 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f mem 0xffaffc00-0xffafffff irq 23 at device 31.2 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: ata2: <ATA channel> at channel 0 on atapci1 Jun 12 05:35:46 <kern.crit> typhon kernel: ata3: <ATA channel> at channel 1 on atapci1 Jun 12 05:35:46 <kern.crit> typhon kernel: ichsmb0: <Intel 82801GB (ICH7) SMBus controller> port 0x400-0x41f at device 31.3 on pci0 Jun 12 05:35:46 <kern.crit> typhon kernel: smbus0: <System Management Bus> on ichsmb0 Jun 12 05:35:46 <kern.crit> typhon kernel: smb0: <SMBus generic I/O> on smbus0 Jun 12 05:35:46 <kern.crit> typhon kernel: acpi_button0: <Power Button> on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0 Jun 12 05:35:46 <kern.crit> typhon kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0 Jun 12 05:35:46 <kern.crit> typhon kernel: atkbd0: [GIANT-LOCKED] Jun 12 05:35:46 <kern.crit> typhon kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 Jun 12 05:35:46 <kern.crit> typhon kernel: ichwd0 on isa0 Jun 12 05:35:46 <kern.crit> typhon kernel: orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcd800-0xcdfff pnpid ORM0000 on isa0 Jun 12 05:35:46 <kern.crit> typhon kernel: sc0: <System console> at flags 0x100 on isa0 Jun 12 05:35:46 <kern.crit> typhon kernel: sc0: VGA <9 virtual consoles, flags=0x300> Jun 12 05:35:46 <kern.crit> typhon kernel: vga0: <Generic ISA VGA> at port 0x3b0-0x3df iomem 0xa0000-0xbffff on isa0 Jun 12 05:35:46 <kern.crit> typhon kernel: coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Jun 12 05:35:46 <kern.crit> typhon kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0 Jun 12 05:35:46 <kern.crit> typhon kernel: est0: Guessed bus clock (high) of 267 MHz Jun 12 05:35:46 <kern.crit> typhon kernel: est0: Guessed high setting of 3071 MHz @ 1292 mV Jun 12 05:35:46 <kern.crit> typhon kernel: est0: Guessed low setting of 1602 MHz @ 1052 mV Jun 12 05:35:46 <kern.crit> typhon kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0 Jun 12 05:35:46 <kern.crit> typhon kernel: coretemp1: <CPU On-Die Thermal Sensors> on cpu1 Jun 12 05:35:46 <kern.crit> typhon kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1 Jun 12 05:35:46 <kern.crit> typhon kernel: est1: Guessed bus clock (high) of 267 MHz Jun 12 05:35:46 <kern.crit> typhon kernel: est1: Guessed high setting of 3071 MHz @ 1292 mV Jun 12 05:35:46 <kern.crit> typhon kernel: est1: Guessed low setting of 1602 MHz @ 1052 mV Jun 12 05:35:46 <kern.crit> typhon kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1 Jun 12 05:35:46 <kern.crit> typhon kernel: Timecounters tick every 10.000 msec Jun 12 05:35:46 <kern.crit> typhon kernel: Expensive timeout(9) function: 0xc04d8900(0xc4d8e800) 0.123575412 s Jun 12 05:35:46 <kern.crit> typhon kernel: da10 at twa0 bus 0 scbus1 target 0 lun 0 Jun 12 05:35:46 <kern.crit> typhon kernel: da10: <AMCC 9690SA-8I DISK 4.10> Fixed Direct Access SCSI-5 device Jun 12 05:35:46 <kern.crit> typhon kernel: da10: 100.000MB/s transfers Jun 12 05:35:46 <kern.crit> typhon kernel: da10: 2860962MB (5859250176 512 byte sectors: 255H 63S/T 364721C) Jun 12 05:35:46 <kern.crit> typhon kernel: SMP: AP CPU #1 Launched! Jun 12 05:35:46 <kern.crit> typhon kernel: Timecounter "TSC-low" frequency 1535994291 Hz quality 1000 Jun 12 05:35:46 <kern.crit> typhon kernel: WARNING: WITNESS option enabled, expect reduced performance. Jun 12 05:35:46 <kern.crit> typhon kernel: WARNING: DIAGNOSTIC option enabled, expect reduced performance. Jun 12 05:35:46 <kern.crit> typhon kernel: da1 at ahc0 bus 0 scbus0 target 1 lun 0 Jun 12 05:35:46 <kern.crit> typhon kernel: da1: <IBM DDRS-39130W S97B> Fixed Direct Access SCSI-2 device Jun 12 05:35:46 <kern.crit> typhon kernel: da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) Jun 12 05:35:46 <kern.crit> typhon kernel: da1: Command Queueing enabled Jun 12 05:35:46 <kern.crit> typhon kernel: da1: 8715MB (17850000 512 byte sectors: 255H 63S/T 1111C) Jun 12 05:35:46 <kern.crit> typhon kernel: da0 at ahc0 bus 0 scbus0 target 0 lun 0 Jun 12 05:35:46 <kern.crit> typhon kernel: da0: <IBM DDRS-39130W S92A> Fixed Direct Access SCSI-2 device Jun 12 05:35:46 <kern.crit> typhon kernel: da0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) Jun 12 05:35:46 <kern.crit> typhon kernel: da0: Command Queueing enabled Jun 12 05:35:46 <kern.crit> typhon kernel: da0: 8715MB (17850000 512 byte sectors: 255H 63S/T 1111C) Jun 12 05:35:46 <kern.crit> typhon kernel: Trying to mount root from ufs:/dev/da0s1a [noatime,rw]... Jun 12 05:35:46 <kern.crit> typhon kernel: WARNING: / was not properly dismounted --- LOR after reboot --- Jun 12 05:38:09 <kern.crit> typhon kernel: done. Jun 12 05:38:11 <kern.crit> typhon kernel: lock order reversal: Jun 12 05:38:11 <kern.crit> typhon kernel: 1st 0xde886080 bufwait (bufwait) @ /src/src-9/sys/kern/vfs_bio.c:2725 Jun 12 05:38:11 <kern.crit> typhon kernel: 2nd 0xc4fdc0dc snaplk (snaplk) @ /src/src-9/sys/ufs/ffs/ffs_snapshot.c:2296 Jun 12 05:38:11 <kern.crit> typhon kernel: 3rd 0xde886328 bufwait (bufwait) @ /src/src-9/sys/kern/vfs_bio.c:2725 Jun 12 05:38:11 <kern.crit> typhon kernel: KDB: stack backtrace: Jun 12 05:38:11 <kern.crit> typhon kernel: db_trace_self_wrapper(c0815bdd,2f6e7265,5f736676,2e6f6962,37323a63,...) at db_trace_self_wrapper+0x26/frame 0xdebd23e0 Jun 12 05:38:11 <kern.crit> typhon kernel: kdb_backtrace(c063659b,c08198cd,c09e9a40,aa5,debd24ac,...) at kdb_backtrace+0x2a/frame 0xdebd243c Jun 12 05:38:11 <kern.crit> typhon kernel: _witness_debugger(c08198cd,de886328,c081fa43,c4c26140,c081ee1e,...) at _witness_debugger+0x25/frame 0xdebd2454 Jun 12 05:38:11 <kern.crit> typhon kernel: witness_checkorder(de886328,9,c081ee1e,aa5,0,...) at witness_checkorder+0x86f/frame 0xdebd24ac Jun 12 05:38:11 <kern.crit> typhon kernel: __lockmgr_args(de886328,81900,c510f1c8,c081f99f,60,...) at __lockmgr_args+0x829/frame 0xdebd2564 Jun 12 05:38:11 <kern.crit> typhon kernel: getblk(c510f11c,fffff7f3,ffffffff,4000,0,...) at getblk+0x154/frame 0xdebd25b8 Jun 12 05:38:11 <kern.crit> typhon kernel: breadn_flags(c510f11c,fffff7f3,ffffffff,4000,0,...) at breadn_flags+0x43/frame 0xdebd25e8 Jun 12 05:38:11 <kern.crit> typhon kernel: bread(c510f11c,fffff7f3,ffffffff,4000,0,...) at bread+0x54/frame 0xdebd2618 Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_balloc_ufs2(c510f11c,918f8000,1,4000,c4c0fb00,...) at ffs_balloc_ufs2+0x16ee/frame 0xdebd27c4 Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_copyonwrite(c509cd50,de886020,de886020,0,0,...) at ffs_copyonwrite+0x3ee/frame 0xdebd2840 Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_geom_strategy(c509cdfc,de886020,0,de886020,3231b8,...) at ffs_geom_strategy+0xeb/frame 0xdebd285c Jun 12 05:38:11 <kern.crit> typhon kernel: bufwrite(de886020,0,c0835390,832,0) at bufwrite+0x169/frame 0xdebd287c Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_bufwrite(de886020,c5118600,100,4000,0,...) at ffs_bufwrite+0x290/frame 0xdebd28a0 Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_update(c509d6a8,1,c0835afd,152,0,...) at ffs_update+0x395/frame 0xdebd2910 Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_syncvnode(c509d6a8,1,0,4c6,c509d754,...) at ffs_syncvnode+0x591/frame 0xdebd296c Jun 12 05:38:11 <kern.crit> typhon kernel: ffs_fsync(debd29bc,4ce,c08219ca,0,debd29cc,...) at ffs_fsync+0x2f/frame 0xdebd2994 Jun 12 05:38:11 <kern.crit> typhon kernel: VOP_FSYNC_APV(c0882560,debd29bc,c088ffc0,c509d6a8,1,...) at VOP_FSYNC_APV+0xa5/frame 0xdebd29ac Jun 12 05:38:11 <kern.crit> typhon kernel: bufsync(c509d754,1,c08219ca,4ce,c503ba00,...) at bufsync+0x38/frame 0xdebd29cc Jun 12 05:38:11 <kern.crit> typhon kernel: bufobj_invalbuf(c509d754,1,0,0,debd2c24,...) at bufobj_invalbuf+0xb5/frame 0xdebd2a04 Jun 12 05:38:11 <kern.crit> typhon kernel: vinvalbuf(c509d6a8,1,0,0,0,...) at vinvalbuf+0x2b/frame 0xdebd2a1c Jun 12 05:38:11 <kern.crit> typhon kernel: vfs_donmount(c506e300,5,0,c515ae80,c515ae80,...) at vfs_donmount+0xabe/frame 0xdebd2c24 Jun 12 05:38:11 <kern.crit> typhon kernel: sys_nmount(c506e300,debd2ccc,c0846650,c081a5d6,286,...) at sys_nmount+0x65/frame 0xdebd2c48 Jun 12 05:38:11 <kern.crit> typhon kernel: syscall(debd2d08) at syscall+0x24d/frame 0xdebd2cfc Jun 12 05:38:11 <kern.crit> typhon kernel: Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xdebd2cfc Jun 12 05:38:11 <kern.crit> typhon kernel: --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x180d20c7, esp = 0xbfbfcdbc, ebp = 0xbfbfd318 --- Unfortunately, I had to disable the cronjob now since people started complaining that the box didn't survive a single night ;-). Thanks, -Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130616063942.GA72803>