Date: Thu, 28 Apr 2011 16:00:23 GMT From: "Helmut Schneider" <jumper99@gmx.de> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/156691: panic when using hard disks as RAW devices within VMWare ESXi 4.1-u1 Message-ID: <201104281600.p3SG0NrP038439@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/156691; it has been noted by GNATS.
From: "Helmut Schneider" <jumper99@gmx.de>
To: <bug-followup@FreeBSD.org>
Cc:
Subject: Re: kern/156691: panic when using hard disks as RAW devices within VMWare ESXi 4.1-u1
Date: Thu, 28 Apr 2011 17:59:33 +0200
OK, I managed to get a backtrace. As a sidenote, I know that the mangled
entry might be caused by hardware failures but
- as said before, Windows 7, Ubuntu and OpenBSD do fine with the same
configuration
- The system is equiped with FBDIMM (ECC) and neither memtest nor the ECC
check report errors
- I tested 3 different disks
- While FreeBSD crashes 4 more VMs continue to run on the same ESXi server,
2 of them FreeBSD 8.2 on virtuak disks
BSDTest# kgdb kernel.debug /root/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: ufs_dirbad: /mnt/da1: bad dir ino 117760 at offset 257: mangled entry
cpuid = 0
KDB: stack backtrace:
#0 0xc08e0d07 at kdb_backtrace+0x47
#1 0xc08b1dc7 at panic+0x117
#2 0xc0af9c56 at ufs_dirbad+0x46
#3 0xc0afae21 at ufs_lookup_+0x371
#4 0xc0afb5ee at ufs_lookup+0x1e
#5 0xc0c02c62 at VOP_CACHEDLOOKUP_APV+0x42
#6 0xc092b056 at vfs_cache_lookup+0xd6
#7 0xc0c04a46 at VOP_LOOKUP_APV+0x46
#8 0xc093290e at lookup+0x69e
#9 0xc093394b at namei+0x63b
#10 0xc094b376 at vn_open_cred+0x96
#11 0xc094b96b at vn_open+0x3b
#12 0xc0948e15 at kern_openat+0x125
#13 0xc09491f5 at kern_open+0x35
#14 0xc0949230 at open+0x30
#15 0xc08eca39 at syscallenter+0x329
#16 0xc0be4e14 at syscall+0x34
#17 0xc0bcbf21 at Xint0x80_syscall+0x21
Uptime: 8m29s
Physical memory: 1011 MB
Dumping 157 MB: 142 126 110 94 78 62 46 30 14
#0 doadump () at pcpu.h:231
231 __asm("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:231
#1 0xc08b1b63 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2 0xc08b1e00 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:592
#3 0xc0af9c56 in ufs_dirbad (ip=0x0, offset=257, how=0xc0cd1a30 "mangled
entry")
at /usr/src/sys/ufs/ufs/ufs_lookup.c:775
#4 0xc0afae21 in ufs_lookup_ (vdp=0xc4c67430, vpp=0xe6879b90,
cnp=0xe6879ba4, dd_ino=0x0)
at /usr/src/sys/ufs/ufs/ufs_lookup.c:395
#5 0xc0afb5ee in ufs_lookup (ap=0xe6879944) at
/usr/src/sys/ufs/ufs/ufs_lookup.c:218
#6 0xc0c02c62 in VOP_CACHEDLOOKUP_APV (vop=0xc0dadee0, a=0xe6879944) at
vnode_if.c:187
#7 0xc092b056 in vfs_cache_lookup (ap=0xe68799cc) at vnode_if.h:80
#8 0xc0c04a46 in VOP_LOOKUP_APV (vop=0xc0dae380, a=0xe68799cc) at
vnode_if.c:123
#9 0xc093290e in lookup (ndp=0xe6879b78) at vnode_if.h:54
#10 0xc093394b in namei (ndp=0xe6879b78) at
/usr/src/sys/kern/vfs_lookup.c:269
#11 0xc094b376 in vn_open_cred (ndp=0xe6879b78, flagp=0xe6879c2c, cmode=384,
vn_open_flags=0,
cred=0xc4913700, fp=0xc495a930) at /usr/src/sys/kern/vfs_vnops.c:130
#12 0xc094b96b in vn_open (ndp=0xe6879b78, flagp=0xe6879c2c, cmode=384,
fp=0xc495a930)
at /usr/src/sys/kern/vfs_vnops.c:94
#13 0xc0948e15 in kern_openat (td=0xc491bb40, fd=-100,
path=0x805b72c <Address 0x805b72c out of bounds>, pathseg=UIO_USERSPACE,
flags=1538, mode=384)
at /usr/src/sys/kern/vfs_syscalls.c:1083
#14 0xc09491f5 in kern_open (td=0xc491bb40, path=0x805b72c <Address
0x805b72c out of bounds>,
pathseg=UIO_USERSPACE, flags=1537, mode=384) at
/usr/src/sys/kern/vfs_syscalls.c:1039
#15 0xc0949230 in open (td=0xc491bb40, uap=0xe6879cec) at
/usr/src/sys/kern/vfs_syscalls.c:1015
#16 0xc08eca39 in syscallenter (td=0xc491bb40, sa=0xe6879ce4) at
/usr/src/sys/kern/subr_trap.c:315
#17 0xc0be4e14 in syscall (frame=0xe6879d28) at
/usr/src/sys/i386/i386/trap.c:1061
#18 0xc0bcbf21 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:264
#19 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) list
264 call syscall
265 add $4, %esp
266 MEXITCOUNT
267 jmp doreti
268
269 ENTRY(fork_trampoline)
270 pushl %esp /* trapframe pointer */
271 pushl %ebx /* arg1 */
272 pushl %esi /* function */
273 call fork_exit
(kgdb)
After the reboot the (GPT partitioned) disk looks like this:
BSDTest# gpart show da1
=> 34 156301421 da1 GPT (75G)
34 128 1 freebsd-boot (64K)
162 4194304 2 freebsd-ufs (2.0G)
4194466 2097152 3 freebsd-swap (1.0G)
6291618 16777216 4 freebsd-ufs (8.0G)
23068834 4194304 5 freebsd-ufs (2.0G)
27263138 129038317 6 freebsd-ufs (62G)
BSDTest# fsck_ufs /dev/da1p1
** /dev/da1p1
CANNOT READ BLK: 128
CONTINUE? [yn] y
THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129, 130, 131, 132, 133,
134, 135, 136, 137, 138, 139, 140, 141, 142, 143,
ioctl (GCINFO): Inappropriate ioctl for device
fsck_ufs: /dev/da1p1: can't read disk label
BSDTest# fsck_ufs /dev/da1p2
** /dev/da1p2
** Last Mounted on /mnt/da1
** Phase 1 - Check Blocks and Sizes
CYLINDER GROUP 1: BAD MAGIC NUMBER
REBUILD CYLINDER GROUP? [yn] y
CYLINDER GROUP 3: BAD MAGIC NUMBER
REBUILD CYLINDER GROUP? [yn] y
UNKNOWN FILE TYPE I=70656
CLEAR? [yn] y
UNKNOWN FILE TYPE I=70657
CLEAR? [yn] y
UNKNOWN FILE TYPE I=70658
CLEAR? [yn] ^C
***** FILE SYSTEM STILL DIRTY *****
BSDTest# fsck_ufs /dev/da1p3
** /dev/da1p3
** Last Mounted on
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 2 used, 506485 free (21 frags, 63308 blocks, 0.0% fragmentation)
***** FILE SYSTEM IS CLEAN *****
BSDTest# fsck_ufs /dev/da1p4
** /dev/da1p4
** Last Mounted on /mnt/da1/usr
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 2 used, 4061061 free (21 frags, 507630 blocks, 0.0% fragmentation)
***** FILE SYSTEM MARKED CLEAN *****
BSDTest# fsck_ufs /dev/da1p5
** /dev/da1p5
** Last Mounted on /mnt/da1/var
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 2 used, 1013013 free (21 frags, 126624 blocks, 0.0% fragmentation)
***** FILE SYSTEM MARKED CLEAN *****
BSDTest# fsck_ufs /dev/da1p6
** /dev/da1p6
** Last Mounted on /mnt/da1/tmp
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 2 used, 31244254 free (22 frags, 3905529 blocks, 0.0%
fragmentation)
***** FILE SYSTEM MARKED CLEAN *****
BSDTest#
And here comes the dmesg:
BSDTest# dmesg -a
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE-p1 #0: Thu Apr 28 15:23:57 UTC 2011
root@BSDTest.charlieroot.de:/usr/obj/usr/src/sys/GENERIC i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2574.73-MHz 686-class
CPU)
Origin = "GenuineIntel" Id = 0x10676 Family = 6 Model = 17 Stepping =
6
Features=0xfe3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,<b31>>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 1073741824 (1024 MB)
avail memory = 1036292096 (988 MB)
ACPI APIC Table: <PTLTD APIC >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x10d0-0x10df mem
0xd4000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem
0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0x2000-0x203f
mem 0xd8920000-0xd893ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:b6:be:a1
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid
ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
Timecounter "TSC" frequency 2574732390 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata1-master UDMA33
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 6.600MB/s transfers (16bit)
da0: Command Queueing enabled
da0: 16384MB (33554432 512 byte sectors: 255H 63S/T 2088C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <ATA ST380817AS 3.42> Fixed Direct Access SCSI-5 device
da1: 6.600MB/s transfers (16bit)
da1: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C)
Trying to mount root from ufs:/dev/da0s1a
Setting hostuuid: 564d5d6b-3d26-341a-4124-f977dfb6bea1.
Setting hostid: 0x0e78eea9.
Entropy harvesting:
interrupts
ethernet
point_to_point
kickstart
.
Starting file system checks:
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 768087 free (3655 frags, 95554 blocks, 0.4%
fragmentation)
/dev/da0s1g: DEFER FOR BACKGROUND CHECKING
/dev/da0s1f: DEFER FOR BACKGROUND CHECKING
/dev/da0s1d: DEFER FOR BACKGROUND CHECKING
/dev/da0s1e: DEFER FOR BACKGROUND CHECKING
Mounting local file systems:
WARNING: /home was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
.
Setting hostname: BSDTest.charlieroot.de
.
Starting Network: lo0 em0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:b6:be:a1
inet6 fe80::20c:29ff:feb6:bea1%em0 prefixlen 64 tentative scopeid
0x1
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
net.inet6.ip6.forwarding:
0
->
0
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
net.inet6.ip6.accept_rtadv:
0
->
1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
IPv4 mapped IPv6 address support=NO
Starting devd.
DHCPREQUEST on em0 to 255.255.255.255 port 67
DHCPACK from 192.168.124.254
bound to 192.168.124.69 -- renewal in 14400 seconds.
Creating and/or trimming log files
.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Clearing /tmp (X related).
Updating motd:
.
Configuring syscons:
keymap
blanktime
.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Thu Apr 28 15:41:34 UTC 2011
Apr 28 15:42:38 BSDTest savecore: reboot after panic: ufs_dirbad: /mnt/da1:
bad dir ino 117760 at offset 257: mangled entry
BSDTest# uname -a
FreeBSD BSDTest.charlieroot.de 8.2-RELEASE-p1 FreeBSD 8.2-RELEASE-p1 #0: Thu
Apr 28 15:23:57 UTC 2011
root@BSDTest.charlieroot.de:/usr/obj/usr/src/sys/GENERIC i386
BSDTest#
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201104281600.p3SG0NrP038439>