Date: Fri, 22 Jun 2007 21:05:16 GMT From: Andrew Yochum <ayochum@pair.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/113957: gmirror is intermittently reporting a degraded mirror array upon reboot. Message-ID: <200706222105.l5ML5GP9076352@www.freebsd.org> Resent-Message-ID: <200706222110.l5MLA30m078329@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 113957 >Category: kern >Synopsis: gmirror is intermittently reporting a degraded mirror array upon reboot. >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Jun 22 21:10:02 GMT 2007 >Closed-Date: >Last-Modified: >Originator: Andrew Yochum >Release: 6.2-RELEASE-p1 >Organization: pair Networks >Environment: FreeBSD 6.2-RELEASE-p1 FreeBSD 6.2-RELEASE-p1 #1: Mon May 7 14:30:09 EDT 2007 support@pair.com:/usr/obj/usr/src/sys/6PAIRc-SMP i386 >Description: Overview: It has been observed that gmirror is intermittently (roughly 20% of the time) reporting the first volume of a mirror as degraded upon reboot. We were able to predictably reproduce the behavior on two different FreeBSD 6.2 machines using different types of hard disks (SCSI and SATA). Files: Attached to this report are two dmesg samples (one from each test machine) that display the full startup information for one of the degraded reboot attempts. dmesg_machine1.txt -10-disk SCSI mirror + stripe array dmesg_machine2.txt -2-disk SATA mirror array >How-To-Repeat: Tests: On the first machine, we originally had set up a stripe + mirror array composed of 10 SCSI hard disks. This was the setup that first brought the odd behavior to our attention. When a degraded array would occur, it was always the same stripe device that needed to be rebuilt (In this case, stripe1). This suggested that either we had some kind of hardware failure on that stripe or something else was occurring to cause the device to be unstable. This was the setup procedure used to build the stripe + mirror array: ----------------------------------------------------------------------------- gstripe label -v -s 1024 stripe1 /dev/da0 /dev/da1 /dev/da2 /dev/da3 /dev/da4 gstripe label -v -s 1024 stripe2 /dev/da5 /dev/da6 /dev/da7 /dev/da8 /dev/da9 gmirror label -v -b load raid1 /dev/stripe/stripe1 /dev/stripe/stripe2 gmirror load newfs /dev/mirror/raid1 mkdir /mirror mount /dev/mirror/raid1 /mirror # Add the following to /boot/loader.conf geom_stripe_load="YES" geom_mirror_load="YES" # And the mount to /etc/fstab /dev/mirror/raid1 /mirror ufs rw 2 2 ----------------------------------------------------------------------------- Though it does not indicate the reason, here is the GEOM portion of the dmesg dump reporting that stripe1 is being rebuilt: -------------------------------------------------------------- GEOM_STRIPE: Device stripe1 created (id=127607522). GEOM_STRIPE: Disk da0 attached to stripe1. GEOM_STRIPE: Disk da1 attached to stripe1. GEOM_STRIPE: Disk da2 attached to stripe1. GEOM_STRIPE: Disk da3 attached to stripe1. GEOM_STRIPE: Disk da4 attached to stripe1. GEOM_STRIPE: Device stripe1 activated. GEOM_STRIPE: Device stripe2 created (id=2058765235). GEOM_STRIPE: Disk da5 attached to stripe2. GEOM_STRIPE: Disk da6 attached to stripe2. GEOM_STRIPE: Disk da7 attached to stripe2. GEOM_STRIPE: Disk da8 attached to stripe2. GEOM_STRIPE: Disk da9 attached to stripe2. GEOM_STRIPE: Device stripe2 activated. GEOM_MIRROR: Device raid1 created (id=702664467). GEOM_MIRROR: Device raid1: provider stripe/stripe1 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe2 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe2 activated. GEOM_MIRROR: Device raid1: provider mirror/raid1 launched. GEOM_MIRROR: Device raid1: rebuilding provider stripe/stripe1. -------------------------------------------------------------- Usually, a successful reboot would report: 'provider stripe/stripe1 activated' right after stripe2 is activated. But as seen in the above case, that line is obviously missing. Instead it reports the rebuild of stripe1. Our first test was to see if perhaps we had some sort of hardware failure. We did this by tearing down the array and swapping the devices that defined the two stripes. If there was a failing hard disk, stripe2 would have exhibited the same symptoms rather than stripe1. Setup procedure with swaped stripe devices: ----------------------------------------------------------------------------- gstripe label -v -s 1024 stripe2 /dev/da0 /dev/da1 /dev/da2 /dev/da3 /dev/da4 gstripe label -v -s 1024 stripe1 /dev/da5 /dev/da6 /dev/da7 /dev/da8 /dev/da9 gmirror label -v -b load raid1 /dev/stripe/stripe1 /dev/stripe/stripe2 ----------------------------------------------------------------------------- Unfortunately, stripe1 continued to display the symptoms; signifying that the problem was not localized to a list of devices and probably not a bad hard disk. The dmesg dump from this attempt: -------------------------------------------------------------- GEOM_STRIPE: Device stripe2 created (id=3454907582). GEOM_STRIPE: Disk da0 attached to stripe2. GEOM_STRIPE: Disk da1 attached to stripe2. GEOM_STRIPE: Disk da2 attached to stripe2. GEOM_STRIPE: Disk da3 attached to stripe2. GEOM_STRIPE: Disk da4 attached to stripe2. GEOM_STRIPE: Device stripe2 activated. GEOM_STRIPE: Device stripe1 created (id=3886319334). GEOM_STRIPE: Disk da5 attached to stripe1. GEOM_STRIPE: Disk da6 attached to stripe1. GEOM_STRIPE: Disk da7 attached to stripe1. GEOM_STRIPE: Disk da8 attached to stripe1. GEOM_STRIPE: Disk da9 attached to stripe1. GEOM_STRIPE: Device stripe1 activated. GEOM_MIRROR: Device raid1 created (id=2591439382). GEOM_MIRROR: Device raid1: provider stripe/stripe2 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe1 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe2 activated. GEOM_MIRROR: Device raid1: provider mirror/raid1 launched. GEOM_MIRROR: Device raid1: rebuilding provider stripe/stripe1. -------------------------------------------------------------- Interesting point to note: The order in which gmirror detects the stripe devices seems to resemble the order in which they are built by gstripe, but the order in which they are 'activated' by gmirror remains the same as in the first test. This prompted a 2nd test case... Our 2nd test involved actually swapping the order in which we listed the stripes in setting up the mirror. Keeping the same stripe configuration as in our 1st test, we tried: ----------------------------------------------------------------------------- gstripe label -v -s 1024 stripe2 /dev/da0 /dev/da1 /dev/da2 /dev/da3 /dev/da4 gstripe label -v -s 1024 stripe1 /dev/da5 /dev/da6 /dev/da7 /dev/da8 /dev/da9 gmirror label -v -b load raid1 /dev/stripe/stripe2 /dev/stripe/stripe1 ----------------------------------------------------------------------------- This time, stripe2 began to exhibit the symptoms instead of stripe1. This indicated that the problem could, in some way, be related to the order in which the mirror devices are defined at the time of setup. It can be seen in the dmesg dump that gmirror now activates stripe1 first instead of stripe2: -------------------------------------------------------------- GEOM_STRIPE: Device stripe2 created (id=1700277137). GEOM_STRIPE: Disk da0 attached to stripe2. GEOM_STRIPE: Disk da1 attached to stripe2. GEOM_STRIPE: Disk da2 attached to stripe2. GEOM_STRIPE: Disk da3 attached to stripe2. GEOM_STRIPE: Disk da4 attached to stripe2. GEOM_STRIPE: Device stripe2 activated. GEOM_STRIPE: Device stripe1 created (id=1012042920). GEOM_STRIPE: Disk da5 attached to stripe1. GEOM_STRIPE: Disk da6 attached to stripe1. GEOM_STRIPE: Disk da7 attached to stripe1. GEOM_STRIPE: Disk da8 attached to stripe1. GEOM_STRIPE: Disk da9 attached to stripe1. GEOM_STRIPE: Device stripe1 activated. GEOM_MIRROR: Device raid1 created (id=3479269816). GEOM_MIRROR: Device raid1: provider stripe/stripe2 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe1 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe1 activated. GEOM_MIRROR: Device raid1: provider mirror/raid1 launched. GEOM_MIRROR: Device raid1: rebuilding provider stripe/stripe2. -------------------------------------------------------------- For our 3rd test, we tried to set up a mirror array with plain hard drive devices instead of stripes. This was to see if perhaps the stripes themselves were actually causing the problem. Setup procedure: ----------------------------------------------------------------------------- gmirror label -v -b load raid1 /dev/da0 /dev/da1 gmirror load newfs /dev/mirror/raid1 mkdir /mirror mount /dev/mirror/raid1 /mirror ----------------------------------------------------------------------------- Unfortunately, this was not the case and the mirror continued to intermittently degrade upon reboot tests. dmesg dump: ---------------------------------------------------------- GEOM_MIRROR: Device raid1 created (id=2593903422). GEOM_MIRROR: Device raid1: provider da0 detected. GEOM_MIRROR: Device raid1: provider da1 detected. GEOM_MIRROR: Device raid1: provider da1 activated. GEOM_MIRROR: Device raid1: provider mirror/raid1 launched. GEOM_MIRROR: Device raid1: rebuilding provider da0. ---------------------------------------------------------- As a final test to rule out any hardware architecture problems, we set up a completely new server using the same FreeBSD kernel and GEOM verions as before, only this time the hard drives were SATA instead of SCSI. The setup procedure: --------------------------------------------------- %gmirror label -v -b load mirror1 /dev/ad5 /dev/ad7 %gmirror load mkdir /mount mount /dev/mirror/mirror1 /mirror # /boot/loader.conf: geom_mirror_load="YES" # /etc/fstab: /dev/mirror/mirror1 /mirror ufs rw 2 2 --------------------------------------------------- As on the previous machine, the first device defined in the mirror at setup time ('/dev/ad5') caused the degraded array: -------------------------------------------------------------- GEOM_MIRROR: Device mirror1 created (id=79998820). GEOM_MIRROR: Device mirror1: provider ad5 detected. GEOM_MIRROR: Device mirror1: provider ad7 detected. GEOM_MIRROR: Device mirror1: provider ad7 activated. GEOM_MIRROR: Device mirror1: provider mirror/mirror1 launched. GEOM_MIRROR: Device mirror1: rebuilding provider ad5. -------------------------------------------------------------- As before, we reversed the order of devices on the mirror to confirm the behavior: --------------------------------------------------- %gmirror label -v -b load mirror1 /dev/ad7 /dev/ad5 --------------------------------------------------- As expected, '/dev/ad7' caused the degraded array this time. dmesg: -------------------------------------------------------------- GEOM_MIRROR: Device mirror1 created (id=2912844912). GEOM_MIRROR: Device mirror1: provider ad5 detected. GEOM_MIRROR: Device mirror1: provider ad7 detected. GEOM_MIRROR: Device mirror1: provider ad5 activated. GEOM_MIRROR: Device mirror1: provider mirror/mirror1 launched. GEOM_MIRROR: Device mirror1: rebuilding provider ad7. -------------------------------------------------------------- >Fix: Patch attached with submission follows: dmesg dump from one of the failed reboot attempts on the first machine. 10 SCSI drives over a RAID01 stripe+mirror array. This should be read as reference to the findings in geom_writeup1.txt. ------------------------------------------------------------------- Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p1 #1: Mon May 7 14:30:09 EDT 2007 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf25 Stepping = 5 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4400<CNTX-ID,<b14>> Logical CPUs per core: 2 real memory = 4160225280 (3967 MB) avail memory = 4072763392 (3884 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic2 <Version 2.0> irqs 48-71 on motherboard ioapic3 <Version 2.0> irqs 72-95 on motherboard ioapic4 <Version 2.0> irqs 96-119 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <unknown> at device 0.1 (no driver attached) pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pci1: <base peripheral, interrupt controller> at device 28.0 (no driver attached) pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1 pci2: <ACPI PCI bus> on pcib2 pci1: <base peripheral, interrupt controller> at device 30.0 (no driver attached) pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1 pci3: <ACPI PCI bus> on pcib3 em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x3000-0x303f mem 0xfc200000-0xfc21ffff irq 28 at device 2.0 on pci3 em0: Ethernet address: 00:30:48:2a:4a:74 em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x3040-0x307f mem 0xfc220000-0xfc23ffff irq 29 at device 2.1 on pci3 em1: Ethernet address: 00:30:48:2a:4a:75 pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci4: <ACPI PCI bus> on pcib4 pci4: <base peripheral, interrupt controller> at device 28.0 (no driver attached) pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci4 pci5: <ACPI PCI bus> on pcib5 fxp0: <Intel 82550 Pro/100 Ethernet> port 0x4000-0x403f mem 0xfc400000-0xfc400fff,0xfc420000-0xfc43ffff irq 100 at device 2.0 on pci5 miibus0: <MII bus> on fxp0 inphy0: <i82555 10/100 media interface> on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:02:b3:ea:eb:cd pci4: <base peripheral, interrupt controller> at device 30.0 (no driver attached) pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4 pci6: <ACPI PCI bus> on pcib6 ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x5400-0x54ff,0x5000-0x50ff mem 0xfc500000-0xfc501fff irq 76 at device 2.0 on pci6 ahd0: [GIANT-LOCKED] aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x5c00-0x5cff,0x5800-0x58ff mem 0xfc502000-0xfc503fff irq 77 at device 2.1 on pci6 ahd1: [GIANT-LOCKED] aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs pci0: <serial bus, USB> at device 29.0 (no driver attached) pci0: <serial bus, USB> at device 29.1 (no driver attached) pci0: <serial bus, USB> at device 29.2 (no driver attached) pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci7: <ACPI PCI bus> on pcib7 pci7: <display, VGA> at device 1.0 (no driver attached) isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel ICH3 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2060-0x206f at device 31.1 on pci0 ata0: <ATA channel 0> on atapci0 ata1: <ATA channel 1> on atapci0 pci0: <serial bus, SMBus> at device 31.3 (no driver attached) acpi_button0: <Power Button> on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xd2000-0xd37ff,0xe0000-0xe3fff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default Waiting 5 seconds for SCSI devices to settle ad0: 114473MB <WDC WD1200JB-00REA0 20.00K20> at ata0-master UDMA100 ad2: 114473MB <WDC WD1200JB-00REA0 20.00K20> at ata1-master UDMA100 ses0 at ahd0 bus 0 target 6 lun 0 ses0: <SUPER GEM318 0> Fixed Processor SCSI-2 device ses0: 3.300MB/s transfers ses0: SAF-TE Compliant Device ses1 at ahd1 bus 0 target 6 lun 0 ses1: <SUPER GEM318 0> Fixed Processor SCSI-2 device ses1: 3.300MB/s transfers ses1: SAF-TE Compliant Device da5 at ahd1 bus 0 target 0 lun 0 da5: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da5: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da5: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da6 at ahd1 bus 0 target 1 lun 0 da6: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da6: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da6: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da7 at ahd1 bus 0 target 2 lun 0 da7: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da7: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da7: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da8 at ahd1 bus 0 target 3 lun 0 da8: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da8: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da8: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da9 at ahd1 bus 0 target 4 lun 0 da9: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da9: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da9: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da0 at ahd0 bus 0 target 0 lun 0 da0: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da1 at ahd0 bus 0 target 1 lun 0 da1: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da1: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da2 at ahd0 bus 0 target 2 lun 0 da2: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da2: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da3 at ahd0 bus 0 target 3 lun 0 da3: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da3: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da3: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da4 at ahd0 bus 0 target 4 lun 0 da4: <SEAGATE ST373453LC 0006> Fixed Direct Access SCSI-3 device da4: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da4: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! GEOM_STRIPE: Device stripe1 created (id=127607522). GEOM_STRIPE: Disk da0 attached to stripe1. GEOM_STRIPE: Disk da1 attached to stripe1. GEOM_STRIPE: Disk da2 attached to stripe1. GEOM_STRIPE: Disk da3 attached to stripe1. GEOM_STRIPE: Disk da4 attached to stripe1. GEOM_STRIPE: Device stripe1 activated. GEOM_STRIPE: Device stripe2 created (id=2058765235). GEOM_STRIPE: Disk da5 attached to stripe2. GEOM_STRIPE: Disk da6 attached to stripe2. GEOM_STRIPE: Disk da7 attached to stripe2. GEOM_STRIPE: Disk da8 attached to stripe2. GEOM_STRIPE: Disk da9 attached to stripe2. GEOM_STRIPE: Device stripe2 activated. GEOM_MIRROR: Device raid1 created (id=702664467). GEOM_MIRROR: Device raid1: provider stripe/stripe1 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe2 detected. GEOM_MIRROR: Device raid1: provider stripe/stripe2 activated. GEOM_MIRROR: Device raid1: provider mirror/raid1 launched. GEOM_MIRROR: Device raid1: rebuilding provider stripe/stripe1. Trying to mount root from ufs:/dev/ad0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 8 files 1 WARNING: /usr was not properly dismounted WARNING: /var was not properly dismounted WARNING: /hot/ad2 was not properly dismounted WARNING: /hot/ad2/usr was not properly dismounted WARNING: /hot/ad2/var was not properly dismounted >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200706222105.l5ML5GP9076352>