Date: Thu, 23 Jul 2009 15:13:18 GMT From: Royce William <royce@tycho.org> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/137034: lock-order reversal near vfs_bio.c, ufs_dirhash.c Message-ID: <200907231513.n6NFDI1Y096280@www.freebsd.org> Resent-Message-ID: <200907231520.n6NFK0Bp024047@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 137034
>Category: kern
>Synopsis: lock-order reversal near vfs_bio.c, ufs_dirhash.c
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Jul 23 15:20:00 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator: Royce William
>Release: 8.0-BETA2
>Organization:
>Environment:
FreeBSD max.prv.tycho.org 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Wed Jul 15 23:25:30 UTC 2009 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386
>Description:
Unloaded system upgrading from 7.2-SECURITY to 8.0-BETA2 via freebsd-update. Halfway through the second install phase, in which I was running the 8.0-BETA2 kernel but a 7.2-SECURITY userland, I got this LOR:
lock order reversal:
1st 0xc557f310 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2558
2nd 0xc5d82600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper(c0c6baf4,f4b9175c,c08bc995,c08ad6db,c0c6e989,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08ad6db,c0c6e989,c592bef8,c592f500,f4b917b8,...) at kdb_backtrace+0x29
_witness_debugger(c0c6e989,c5d82600,c0c8f767,c592f500,c0c8f400,...) at _witness_debugger+0x25
witness_checkorder(c5d82600,9,c0c8f400,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c5d82600,0,c0c8f400,11d,c5ff3d24,...) at _sx_xlock+0x85
ufsdirhash_acquire(c557f2b0,f4b918d0,16c,e581c8ac,f4b91888,...) at ufsdirhash_acquire+0x35
ufsdirhash_add(c5ff3d24,f4b918d0,8ac,f4b91874,f4b91878,...) at ufsdirhash_add+0x13
ufs_direnter(c5b96648,cceff324,f4b918d0,f4b91bd0,0,...) at ufs_direnter+0x729
ufs_makeinode(f4b91bd0,0,f4b91abc,f4b91a18,c0bb6745,...) at ufs_makeinode+0x508
ufs_create(f4b91abc,f4b91ad4,0,0,f4b91ba4,...) at ufs_create+0x30
VOP_CREATE_APV(c0d6e7e0,f4b91abc,f4b91bd0,f4b91a54,0,...) at VOP_CREATE_APV+0xa5
vn_open_cred(f4b91ba4,f4b91c5c,1a0,0,c8596900,...) at vn_open_cred+0x215
vn_open(f4b91ba4,f4b91c5c,1a0,c78877e0,0,...) at vn_open+0x3b
kern_openat(d872e480,ffffff9c,28303514,0,602,...) at kern_openat+0x11f
kern_open(d872e480,28303514,0,601,1b6,...) at kern_open+0x35
open(d872e480,f4b91cf8,c,d872e480,c0d4d44c,...) at open+0x30
syscall(f4b91d38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281d8833, esp = 0xbfbfe16c, ebp = 0xbfbfe208 ---
The system is using ZFS for some mounts (may not be relevant):
royce@max$ mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
tank/home on /home (zfs, local, nosuid)
tank on /tank (zfs, local)
tank/tmp on /tmp (zfs, local, noatime, nosuid)
tank/usr on /usr (zfs, local, noatime)
tank/var on /var (zfs, local, noatime, nosuid)
royce@max$ zpool upgrade
This system is currently running ZFS pool version 13.
The following pools are out of date, and can be upgraded. After being
upgraded, these pools will no longer be accessible by older software versions.
VER POOL
--- ------------
6 tank
>How-To-Repeat:
Probably I/O related, but in brief:
#### Install FreeBSD 7.2.
# freebsd-update fetch install
# shutdown -r now
# ## System is now running 7.2-SECURITY as of 2009-07-21
#### Upgrade to 8.0-BETA2 kernel.
# freebsd-update -r 8.0-BETA2 upgrade
# freebsd-update install
# shutdown -r now
#### Perform second phase of install.
# freebsd-update install
#### LOR occurred here.
>Fix:
Patch attached with submission follows:
#
# royce@tycho.org 2009-07-22
#
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-BETA2 #0: Wed Jul 15 23:25:30 UTC 2009
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (3000.12-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf43 Stepping = 3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x649d<SSE3,DTES64,MON,DS_CPL,EST,CNXT-ID,CX16,xTPR>
AMD Features=0x20100000<NX,LM>
TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 2077204480 (1980 MB)
ACPI APIC Table: <IntelR AWRDACPI>
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <IntelR AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7f5e0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
ath0: <Atheros 5212> mem 0xd0000000-0xd000ffff irq 24 at device 1.0 on pci2
ath0: [ITHREAD]
ath0: AR2413 mac 7.9 RF2413 phy 4.5
vgapci0: <VGA-compatible display> port 0xe000-0xe007 mem 0xd0400000-0xd047ffff,0xc0000000-0xcfffffff,0xd0480000-0xd04bffff irq 16 at device 2.0 on pci0
agp0: <Intel E7221 SVGA controller> on vgapci0
agp0: detected 7932k stolen memory
agp0: aperture size is 256M
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xd0200000-0xd020ffff irq 18 at device 0.0 on pci4
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:30:48:85:d3:bc
bge0: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xd0300000-0xd030ffff irq 19 at device 0.0 on pci5
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:30:48:85:d3:bd
bge1: [ITHREAD]
uhci0: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-A> port 0xe100-0xe11f irq 23 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-A> on uhci0
uhci1: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-B> port 0xe200-0xe21f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-B> on uhci1
uhci2: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-C> port 0xe300-0xe31f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-C> on uhci2
uhci3: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-D> port 0xe400-0xe41f irq 16 at device 29.3 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus3: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-D> on uhci3
ehci0: <Intel 82801FB (ICH6) USB 2.0 controller> mem 0xd04c0000-0xd04c03ff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 82801FB (ICH6) USB 2.0 controller> on ehci0
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH6 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH6 SATA150 controller> port 0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb07,0xec00-0xec03,0xed00-0xed0f mem 0xd04c3000-0xd04c33ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atrtc0: <AT realtime clock> port 0x70-0x73 irq 8 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
cpu0: <ACPI CPU> on acpi0
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xcc000-0xcffff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: parallel port not found.
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounter "TSC" frequency 3000120615 Hz quality 800
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ZFS NOTICE: prefetch is disabled by default on i386 - add enable to tunable to change.
ZFS filesystem version 13
ZFS storage pool version 13
ad4: 239372MB <Maxtor 7L250S0 BANC1E00> at ata2-master SATA150
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
ad6: 239372MB <Maxtor 7L250S0 BANC1E00> at ata3-master SATA150
ar0: 239372MB <Intel MatrixRAID RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
WARNING: WITNESS option enabled, expect reduced performance.
GEOM: ar0s1: geometry does not match label (16h,63s != 255h,63s).
Root mount waiting for: usbus4 usbus3 usbus2 usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
Root mount waiting for: usbus4
uhub4: 8 ports with 8 removable, self powered
Trying to mount root from ufs:/dev/ad4s1a
bge0: link state changed to UP
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200907231513.n6NFDI1Y096280>