Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 3 Jun 2009 23:08:00 -0500 (CDT)
From:      Wes Morgan <morganw@chemikals.org>
To:        current@freebsd.org
Subject:   Panic in kern_jail prison_priv_check()
Message-ID:  <alpine.BSF.2.00.0906032225550.1619@ibyngvyr.purzvxnyf.bet>

next in thread | raw e-mail | index | archive | help
I'm getting a reproducible panic trying to access an nfs-exported zfs 
filesystem with a kernel from a few hours ago, as well as one from over 
the weekend. Filesystem mounts ok, but accessing it causes an immediate 
panic.

Looks like possibly a problem between zfs and the new jail code. Neither 
of the zfs functions involved have been changed in a way that would appear 
to affect this, though. Could be wrong! It may not be pertinent but I do 
have vfs.usermount enabled. The value for "priv" in prison_priv_check() is 
334, PRIV_VFS_MOUNT_OWNER. The contents of the "credentials" is:


$6 = {cr_ref = 1, cr_uid = 4294967294, cr_ruid = 0, cr_svuid = 0,
   cr_ngroups = 1, cr_groups = {4294967294, 0, 2, 3, 4, 5, 6, 20, 25, 26,
   31, 0, 0, 0, 0, 0}, cr_rgid = 0, cr_svgid = 0, cr_uidinfo = 0x0,
   cr_ruidinfo = 0x0, cr_prison = 0x0, cr_vimage = 0x0, cr_flags = 0,
   cr_pspare = {0x0, 0x0}, cr_label = 0x0, cr_audit = {ai_auid = 0,
   ai_mask = {am_success = 0, am_failure = 0},
   ai_termid = {at_port = 0, at_type = 0, at_addr = {0, 0, 0, 0}},
   ai_asid = 0, ai_flags = 0}}


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address	= 0x6dc
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff803109e1
stack pointer	        = 0x28:0xffffff80c58de5a0
frame pointer	        = 0x28:0xffffff0006248a00
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1186 (nfsd: service)
trap number		= 12
panic: page fault
cpuid = 2

Syncing disks, vnodes remaining...0 All buffers synced.
Uptime: 1m23s
Physical memory: 8181 MB
Dumping 1662 MB:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15

(kgdb) #0  doadump () at pcpu.h:223
#1  0x0000000000000000 in ?? ()
#2  0xffffffff803335fd in boot (howto=256)
     at /usr/src/sys/kern/kern_shutdown.c:420
#3  0xffffffff803338ed in panic (fmt=Variable "fmt" is not available.
)
     at /usr/src/sys/kern/kern_shutdown.c:576
#4  0xffffffff804fdd68 in trap_fatal (frame=0xffffff80c58de4f0, eva=1756)
     at /usr/src/sys/amd64/amd64/trap.c:852
#5  0xffffffff804fdfcf in trap_pfault (frame=0xffffff80c58de4f0, usermode=0)
     at /usr/src/sys/amd64/amd64/trap.c:768
#6  0xffffffff804fe917 in trap (frame=0xffffff80c58de4f0)
     at /usr/src/sys/amd64/amd64/trap.c:494
#7  0xffffffff804da7e3 in calltrap ()
     at /usr/src/sys/amd64/amd64/exception.S:223
#8  0xffffffff803109e1 in prison_priv_check (cred=0xffffff0006248a00,
     priv=334) at /usr/src/sys/kern/kern_jail.c:3315
#9  0xffffffff80329380 in priv_check_cred (cred=0xffffff0006248a00, priv=334,
     flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_priv.c:93
#10 0xffffffff808bf92e in secpolicy_vnode_access (cred=0xffffff0006248a00,
     vp=0xffffff0006a1d760, owner=Variable "owner" is not available.
)
     at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_policy.c:125
#11 0xffffffff809207f7 in zfs_zaccess (zp=0xffffff0006a1e758, mode=Variable "mode" is not available.
)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c:2435
#12 0xffffffff8093244e in zfs_freebsd_access (ap=Variable "ap" is not available.
)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1067
#13 0xffffffff805203d2 in VOP_ACCESS_APV (vop=Variable "vop" is not available.
) at vnode_if.c:571
#14 0xffffffff804495c2 in nfsrv_access (vp=0xffffff0006a1d760, accmode=128,
     cred=0xffffff0006248a00, rdonly=Variable "rdonly" is not available.
) at vnode_if.h:254
#15 0xffffffff80449c86 in nfsrv3_access (nfsd=0xffffff80c58dea10, slp=Variable "slp" is not available.
)
     at /usr/src/sys/nfsserver/nfs_serv.c:238
#16 0xffffffff8045618b in nfssvc_program (rqst=0xffffff000a366000, xprt=Variable "xprt" is not available.
)
     at /usr/src/sys/nfsserver/nfs_srvkrpc.c:410
#17 0xffffffff8047292a in svc_run_internal (pool=0xffffff00060eda00,
     ismaster=0) at /usr/src/sys/rpc/svc.c:883
#18 0xffffffff80472a9d in svc_thread_start (arg=Variable "arg" is not available.
)
     at /usr/src/sys/rpc/svc.c:1188
#19 0xffffffff8030d347 in fork_exit (
     callout=0xffffffff80472a8f <svc_thread_start>, arg=0xffffff00060eda00,
     frame=0xffffff80c58dec90) at /usr/src/sys/kern/kern_fork.c:829
#20 0xffffffff804dabee in fork_trampoline ()
     at /usr/src/sys/amd64/amd64/exception.S:552


Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #6: Wed Jun  3 19:46:36 CDT 2009
     root@volatile:/usr/obj/usr/src/sys/VOLATILE
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU    Q8400  @ 2.66GHz (2666.68-MHz K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x1067a  Stepping = 10
   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0x408e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant
real memory  = 8589934592 (8192 MB)
avail memory = 8287219712 (7903 MB)
ACPI APIC Table: <090308 APIC1358>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
ioapic0 <Version 2.0> irqs 0-23 on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <090308 XSDT1358> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci7: <ACPI PCI bus> on pcib1
mpt0: <LSILogic SAS/SATA Adapter> port 0xe800-0xe8ff mem 0xfbffc000-0xfbffffff,0xfbfe0000-0xfbfeffff irq 16 at device 0.0 on pci7
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.19.0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci6: <ACPI PCI bus> on pcib3
mpt1: <LSILogic SAS/SATA Adapter> port 0xd800-0xd8ff mem 0xfbbfc000-0xfbbfffff,0xfbbe0000-0xfbbeffff irq 18 at device 6.0 on pci6
mpt1: [ITHREAD]
mpt1: MPI Version=1.5.16.0
mpt1: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt1: 0 Active Volumes (2 Max)
mpt1: 0 Hidden Drive Members (14 Max)
Ambiguous scbus configuration for mpt1 bus 1, cannot wire down.  The kernel config entry for scbus1 should specify a controller bus.
Scbus will be assigned dynamically.
pcib4: <PCI-PCI bridge> mem 0xfb7ffc00-0xfb7ffc7f irq 16 at device 0.1 on pci4
pci5: <PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib5
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem 0xfb6f0000-0xfb6fffff irq 16 at device 0.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:22:15:56:bb:1e
bge0: [ITHREAD]
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib6
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem 0xfb5f0000-0xfb5fffff irq 17 at device 0.0 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:22:15:56:ba:24
bge1: [ITHREAD]
uhci0: <UHCI (generic) USB controller> port 0xac00-0xac1f irq 23 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x0f00
usbus0: <UHCI (generic) USB controller> on uhci0
uhci1: <UHCI (generic) USB controller> port 0xb000-0xb01f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x0f00
usbus1: <UHCI (generic) USB controller> on uhci1
uhci2: <UHCI (generic) USB controller> port 0xb080-0xb09f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x0f00
usbus2: <UHCI (generic) USB controller> on uhci2
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xfb3ff800-0xfb3ffbff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
vgapci0: <VGA-compatible display> port 0xcc00-0xcc7f mem 0xf8000000-0xf9ffffff,0xfb4c0000-0xfb4fffff at device 3.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
atapci1: <Intel ICH7 SATA300 controller> port 0xbc00-0xbc07,0xb880-0xb883,0xb800-0xb807,0xb480-0xb483,0xb400-0xb40f mem 0xfb3ffc00-0xfb3fffff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI v1.10 controller with 4 3Gbps ports, PM not supported
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
cpu0: <ACPI CPU> on acpi0
ACPI Warning (tbutils-0243): Incorrect checksum in table [OEMB] -  91, should be 90 [20070320]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
cpu1: <ACPI CPU> on acpi0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
cpu2: <ACPI CPU> on acpi0
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
cpu3: <ACPI CPU> on acpi0
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ad0: 152627MB <Seagate ST3160815A 3.AAC> at ata0-master UDMA100
ZFS filesystem version 13
ZFS storage pool version 13
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
acd0: DVDR <HL-DT-ST BDDVDRW GGC-H20L/1.03> at ata2-master SATA150
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x40 0x00 0x01
acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x40 0x00 0x01
da0 at mpt0 bus 0 target 0 lun 0
da0: <ATA ST3500641AS J> Fixed Direct Access SCSI-5 device 
da0: 300.000MB/s transfers
da0: Command Queueing Enabled
da0: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da1 at mpt0 bus 0 target 1 lun 0
da1: <ATA ST31500341AS CC1J> Fixed Direct Access SCSI-5 device 
da1: 300.000MB/s transfers
da1: Command Queueing Enabled
da1: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
da2 at mpt0 bus 0 target 2 lun 0
da2: <ATA ST31500341AS CC1H> Fixed Direct Access SCSI-5 device 
da2: 300.000MB/s transfers
da2: Command Queueing Enabled
da2: 1430799MB (2930277168 512 byte sectors: 255H 63S/T 182401C)
da3 at mpt0 bus 0 target 3 lun 0
da3: <ATA WDC WD10EACS-00D 1A01> Fixed Direct Access SCSI-5 device 
da3: 300.000MB/s transfers
da3: Command Queueing Enabled
da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da4 at mpt0 bus 0 target 4 lun 0
da4: <ATA Maxtor 6L300S0 1E00> Fixed Direct Access SCSI-5 device 
da4: 300.000MB/s transfers
da4: Command Queueing Enabled
da4: 286188MB (586114704 512 byte sectors: 255H 63S/T 36483C)
da6 at mpt0 bus 0 target 6 lun 0
da6: <ATA Maxtor 7L300S0 1G20> Fixed Direct Access SCSI-5 device 
da6: 300.000MB/s transfers
da6: Command Queueing Enabled
da6: 286188MB (586114704 512 byte sectors: 255H 63S/T 36483C)
da7 at mpt0 bus 0 target 7 lun 0
da7: <ATA WDC WD10EADS-00L 1A01> Fixed Direct Access SCSI-5 device 
da7: 300.000MB/s transfers
da7: Command Queueing Enabled
da7: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da8 at mpt1 bus 0 target 0 lun 0
da8: <ATA SAMSUNG HD501LJ 0-13> Fixed Direct Access SCSI-5 device 
da8: 300.000MB/s transfers
da8: Command Queueing Enabled
da8: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da9 at mpt1 bus 0 target 1 lun 0
da9: <ATA ST3500320NS SN04> Fixed Direct Access SCSI-5 device 
da9: 300.000MB/s transfers
da9: Command Queueing Enabled
da9: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da10 at mpt1 bus 0 target 2 lun 0
da10: <ATA SAMSUNG HD501LJ 0-10> Fixed Direct Access SCSI-5 device 
da10: 300.000MB/s transfers
da10: Command Queueing Enabled
da10: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da11 at mpt1 bus 0 target 3 lun 0
da11: <ATA ST3500320NS SN04> Fixed Direct Access SCSI-5 device 
da11: 300.000MB/s transfers
da11: Command Queueing Enabled
da11: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da12 at mpt1 bus 0 target 4 lun 0
da12: <ATA WDC WD5000AAKS-0 1C01> Fixed Direct Access SCSI-5 device 
da12: 300.000MB/s transfers
da12: Command Queueing Enabled
da12: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da13 at mpt1 bus 0 target 5 lun 0
da13: <ATA SAMSUNG HD501LJ 0-10> Fixed Direct Access SCSI-5 device 
da13: 300.000MB/s transfers
da13: Command Queueing Enabled
da13: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da14 at mpt1 bus 0 target 6 lun 0
da14: <ATA ST3500630AS E> Fixed Direct Access SCSI-5 device 
da14: 300.000MB/s transfers
da14: Command Queueing Enabled
da14: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
da15 at mpt1 bus 0 target 7 lun 0
da15: <ATA WDC WD5002ABYS-0 3B02> Fixed Direct Access SCSI-5 device 
da15: 300.000MB/s transfers
da15: Command Queueing Enabled
da15: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
uhub3: 6 ports with 6 removable, self powered
ugen0.2: <American Power Conversion> at usbus0
cd0 at ata1 bus 0 target 0 lun 0
cd0: <HL-DT-ST BDDVDRW GGC-H20L 1.03> Removable CD-ROM SCSI-0 device 
cd0: 3.300MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
GEOM: da7: partition 1 does not start on a track boundary.
GEOM: da7: partition 1 does not end on a track boundary.
Trying to mount root from zfs:root



Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address	= 0x6dc
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff803109e1
stack pointer	        = 0x28:0xffffff80c58de5a0
frame pointer	        = 0x28:0xffffff0006248a00
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1186 (nfsd: service)
trap number		= 12
panic: page fault
cpuid = 2

Syncing disks, vnodes remaining...0 All buffers synced.
Uptime: 1m23s
Physical memory: 8181 MB
Dumping 1662 MB:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0906032225550.1619>