Date: Sat, 13 Jul 2013 10:14:06 +0200 From: Ian FREISLICH <ianf@clue.co.za> To: Konstantin Belousov <kostikbel@gmail.com> Cc: freebsd-current@freebsd.org Subject: Re: Filesystem wedges caused by r251446 Message-ID: <E1Uxuy4-0003KB-OZ@clue.co.za> In-Reply-To: <20130713054220.GJ91021@kib.kiev.ua> References: <20130713054220.GJ91021@kib.kiev.ua> <20130712201051.GI91021@kib.kiev.ua> <201307110923.06548.jhb@freebsd.org> <201307091202.24493.jhb@freebsd.org> <E1UufRq-0001sg-HG@clue.co.za> <E1UxEWB-0000il-21@clue.co.za> <E1Uxhoe-0000d9-Sc@clue.co.za> <E1Uxkyo-0002mE-MJ@clue.co.za>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multipart MIME message.
--==_Exmh_1373703160_23330
Content-Type: text/plain; charset=us-ascii
Konstantin Belousov wrote:
> On Fri, Jul 12, 2013 at 11:34:18PM +0200, Ian FREISLICH wrote:
> > (kgdb) print runningbufreq
> > $1 = 1
> > (kgdb) print runningbufspace
> > $2 = 0
> > (kgdb) print lorunningspace
> > $3 = 4587520
> > (kgdb) print hirunningspace
> > $4 = 4194304
>
> This is extremely weird. The hirunningspace is less then lorunningspace,
> am I right ? This causes the runningbufspace machinery to never wake up
Yes. This state of affairs doesn't happen on r251445 and further
testing on my side shows it doesn't hapen on all my amd64 servers.
It appears that this particular server type (Dell R200) running
amd64 with geom_mirror is affected. I will have to test further
by destroying the mirror and removing it from the kernel and see
if I can still reproduce the issue. Perhaps r251446 exposes
insufficient locking on opperations affecting these variables.
FWIW, I cannot reproduce the problem if the mirror is rebuilding.
> I just verified on the 4G VM on amd64, my numbers for lo is 4587520,
> for high 6881280. Verify your tuning and kernel options, which you should
> have provided with the original report, I think.
Sorry about that (and I'm relieved:) I had originally compiled with
CPUTYPE?=opteron which is incorrect for this CPU. However the
problem persists with CPUTYPE?=core2, but I'm not sure how much of
a difference this makes with clang. Also, I have another affected
host that's compiled with gcc and the correct CPUTYPE so I doubt
it's the compiler.
I've attached make.conf, kernelconfig and dmesg.boot. You'll notice
it's r251446M - which is a result of your patch.
Ian
--
Ian Freislich
--==_Exmh_1373703160_23330
Content-Type: text/plain ; name="FIREWALL"; charset=us-ascii
Content-Description: FIREWALL
Content-Disposition: attachment; filename="FIREWALL"
cpu HAMMER
ident "FIREWALL"
options SCHED_ULE
options INET #InterNETworking
options FFS #Berkeley Fast Filesystem
options UFS_ACL #Support for access control lists
options UFS_DIRHASH #Improve performance on big directories
options SOFTUPDATES #Enable FFS soft updates support
options PSEUDOFS #Pseudo-filesystem framework
options PROCFS
options GEOM_PART_GPT
options GEOM_LABEL
options GEOM_MIRROR
options GEOM_GATE # Userland services.
options COMPAT_43
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_FREEBSD32
options COMPAT_FREEBSD4 #Compatible with FreeBSD4
options COMPAT_FREEBSD5 #Compatible with FreeBSD4
options COMPAT_FREEBSD6 #Compatible with FreeBSD4
options COMPAT_FREEBSD7 #Compatible with FreeBSD4
options COMPAT_LINUX32
options LINPROCFS
options LINSYSFS
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options CONSPEED=115200
options PRINTF_BUFR_SIZE=128
device pf
device pflog
device pfsync
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
# Debugging for use in -current
options KDB # Enable kernel debugger support.
options DDB # Support DDB.
options GDB # Support remote GDB.
options KDB_TRACE
options KDB_UNATTENDED
options ALT_BREAK_TO_DEBUGGER
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC
makeoptions DEBUG=-g
# To make an SMP kernel, the next two are needed
options SMP # Symmetric MultiProcessor Kernel
device cpufreq
device acpi
device pci
device smb
device smbus
device ichsmb
# ATA controllers
device mfi
device scbus # SCSI bus (required for ATA/SCSI)
device ahci # AHCI-compatible SATA controllers
device ata
device ada # Direct Access (disks)
device da # Direct Access (disks)
device cd # CD
device pass # Passthrough device (direct ATA/SCSI access)
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device kbdmux
device psm # PS/2 mouse
device vga # VGA video card driver
device sc
device agp # support several AGP chipsets
# Serial (COM) ports
device uart
device smb
device smbus
device ichsmb
device miibus
device bce
device bge
device em
device igb
device vlan
option VLAN_ARRAY
device carp
# Pseudo devices - the number indicates how many units to allocate.
device random # Entropy device
device loop # Network loopback
device ether # Ethernet support
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
device bpf # Berkeley packet filter
device usb
device uhci
device ehci
device ohci
device ums
device ukbd
device ucom
device ulpt
device uplcom
device umass
device uhid
--==_Exmh_1373703160_23330
Content-Type: text/plain ; name="dmesg.boot"; charset=us-ascii
Content-Description: dmesg.boot
Content-Disposition: attachment; filename="dmesg.boot"
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #2 r251446M: Sat Jul 13 09:14:34 SAST 2013
ianf@fw1.smmt.gp-online.net:/usr/obj/usr/src/sys/FIREWALL amd64
FreeBSD clang version 3.3 (trunk 178860) 20130405
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
CPU: Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz (2666.82-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Family = 0x6 Model = 0x17 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x8e39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 3966435328 (3782 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <DELL PE_SC3> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004201> mem 0xdfdf0000-0xdfdfffff irq 16 at device 0.0 on pci3
bge0: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 1000BASE-T media interface> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:25:64:3c:33:8d
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib4
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004201> mem 0xdfef0000-0xdfefffff irq 17 at device 0.0 on pci4
bge1: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 1000BASE-T media interface> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: Ethernet address: 00:25:64:3c:33:8e
uhci0: <Intel 82801I (ICH9) USB controller> port 0xdc60-0xdc7f irq 21 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xdc80-0xdc9f irq 20 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0xdca0-0xdcbf irq 21 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xdfcffc00-0xdfcfffff irq 21 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xec00-0xecff mem 0xd0000000-0xd7ffffff,0xdfff0000-0xdfffffff irq 19 at device 5.0 on pci5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH9 SATA300 controller> port 0xdc30-0xdc37,0xdc28-0xdc2b,0xdc38-0xdc3f,0xdc2c-0xdc2f,0xdc40-0xdc4f,0xdc50-0xdc5f irq 23 at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x90 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xca000-0xcb7ff,0xec000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <Hitachi HUA721010KLA330 GKAOA9N1> ATA-7 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad0
ada1 at ata3 bus 0 scbus1 target 0 lun 0
ada1: <Hitachi HUA721010KLA330 GKAOA9N1> ATA-7 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad1
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1333409705 Hz quality 1000
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
cd0 at ata2 bus 0 scbus0 target 1 lun 0
cd0: <TSSTcorp DVD+-RW TS-L633C D150> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
GEOM: ada0: the secondary GPT header is not in the last LBA.
GEOM: ada1: the secondary GPT header is not in the last LBA.
GEOM_MIRROR: Device mirror/d0 launched (2/2).
Root mount waiting for: usbus3
Root mount waiting for: usbus3
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
ugen3.2: <vendor 0x04b4> at usbus3
uhub4: <vendor 0x04b4 product 0x6560, class 9/0, rev 2.00/0.0b, addr 2> on usbus3
uhub4: MTT enabled
Root mount waiting for: usbus3
uhub4: 4 ports with 4 removable, self powered
Trying to mount root from ufs:/dev/mirror/d0p2 [rw]...
bge0: link state changed to UP
carp: VHID 3@vlan3: INIT -> BACKUP
carp: demoted by -240 to 2400 (interface up)
vlan3: link state changed to UP
carp: VHID 4@vlan4: INIT -> BACKUP
carp: demoted by -240 to 2160 (interface up)
vlan4: link state changed to UP
carp: VHID 5@vlan5: INIT -> BACKUP
carp: demoted by -240 to 1920 (interface up)
vlan5: link state changed to UP
carp: VHID 6@vlan6: INIT -> BACKUP
carp: demoted by -240 to 1680 (interface up)
vlan6: link state changed to UP
carp: VHID 7@vlan7: INIT -> BACKUP
carp: demoted by -240 to 1440 (interface up)
vlan7: link state changed to UP
carp: VHID 8@vlan8: INIT -> BACKUP
carp: demoted by -240 to 1200 (interface up)
vlan8: link state changed to UP
carp: VHID 9@vlan9: INIT -> BACKUP
carp: demoted by -240 to 960 (interface up)
vlan9: link state changed to UP
carp: VHID 12@vlan12: INIT -> BACKUP
carp: demoted by -240 to 720 (interface up)
vlan12: link state changed to UP
carp: VHID 13@vlan13: INIT -> BACKUP
carp: demoted by -240 to 480 (interface up)
vlan13: link state changed to UP
carp: VHID 40@vlan40: INIT -> BACKUP
carp: demoted by -240 to 240 (interface up)
vlan40: link state changed to UP
bge1: link state changed to UP
--==_Exmh_1373703160_23330
Content-Type: text/plain ; name="make.conf"; charset=us-ascii
Content-Description: make.conf
Content-Disposition: attachment; filename="make.conf"
MAKE_IDEA= YES # IDEA (128 bit symmetric encryption)
PRINTERDEVICE= ps
USA_RESIDENT=NO
CPUTYPE?=core2
HAVE_MOTIF= yes
MAKE_KERBEROS5= yes
ENABLE_SUID_K5SU= yes
KERNCONF=FIREWALL
BOOT_COMCONSOLE_SPEED=115200
#WITHOUT_CLANG_IS_CC=yes
WITH_PKGNG= yes
# added by use.perl 2013-06-13 18:12:04
PERL_VERSION=5.14.4
--==_Exmh_1373703160_23330--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1Uxuy4-0003KB-OZ>