Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 15 Oct 2013 22:34:14 +0100 (BST)
From:      Anton Shterenlikht <mexas@bris.ac.uk>
To:        FreeBSD-gnats-submit@freebsd.org
Subject:   kern/183007: ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
Message-ID:  <201310152134.r9FLYE06010287@mech-aslap33.men.bris.ac.uk>
Resent-Message-ID: <201310152140.r9FLe2LZ059707@freefall.freebsd.org>

next in thread | raw e-mail | index | archive | help

>Number:         183007
>Category:       kern
>Synopsis:       ia64: panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 15 21:40:01 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Anton Shterenlikht
>Release:        FreeBSD 10.0-CURRENT amd64
>Organization:
University of Bristol	
>Environment:
System: FreeBSD mech-aslap33.men.bris.ac.uk 10.0-CURRENT FreeBSD 10.0-CURRENT #6 r255093: Sat Aug 31 13:49:24 BST 2013 root@mech-aslap33.men.bris.ac.uk:/usr/obj/usr/src/sys/MINKY amd64


	
>Description:

below are the outputs of ddb commands:

show msgbuf
show proc
run lockinfo
show pcpu
bt
show threads
ps
alltrace
show pginfo


db> show msgbuf
msgbufp = 0xe0000040fffdffb8
magic = 63062, size = 98232, r= 11031, w = 11766, ptr = 0xe0000040fffc8000, cksum= 898967
panic: deadlkres: possible deadlock detected for 0xe0000000128ecd80, blocked for 900060 ticks

cpuid = 1
KDB: stack backtrace:
db_trace_self(0x9ffc000000158380) at db_trace_self+0x40
db_trace_self_wrapper(0x9ffc00000060b900) at db_trace_self_wrapper+0x70
kdb_backtrace(0x9ffc000000ee1b48, 0x9ffc000000592a40, 0x40c, 0x9ffc0000010b48a0) at kdb_backtrace+0xc0
vpanic(0x9ffc000000dc54d8, 0xa00000009de3b518) at vpanic+0x260
panic(0x9ffc000000dc54d8, 0x9ffc000000dc5768, 0xe0000000128ecd80, 0xdbbdc) at panic+0x80
deadlkres(0xdbbdc, 0xe0000000128ecd80, 0x9ffc000000dc8140, 0x9ffc000000dc5468) at deadlkres+0x420
fork_exit(0x9ffc000000e23230, 0x0, 0xa00000009de3b550) at fork_exit+0x120
enter_userland() at enter_userland
KDB: enter: panic
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #7 r255488: Tue Oct 15 10:09:07 BST 2013
    root@mech-as28.men.bris.ac.uk:/usr/obj/usr/src/sys/ZEEV ia64
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
CPU: Madison II (1600 MHz Itanium 2)
  Origin = "GenuineIntel"  Revision = 2
  Features = 0x1<LB>
real memory  = 10737418240 (10240 MB)
avail memory = 10484695040 (9998 MB)
FPSWA Revision = 0x10012, Entry = 0xe0000040ffcc4050
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0: ACPI Id=0, SAPIC Id=0, SAPIC Eid=0 (BSP)
cpu1: ACPI Id=1, SAPIC Id=1, SAPIC Eid=0
random: <Software, Yarrow> initialized
Event timer "ITC" frequency 1600000000 Hz quality 1000
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20130823/tbfadt-603)
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20130823/tbfadt-603)
acpi0: <HP> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> iomem 0xff5c1004-0xff5c1007 on acpi0
acpi_tz0: <Thermal Zone> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pcib0: could not evaluate _ADR - AE_NOT_FOUND
pci0: <ACPI PCI bus> on pcib0
ohci0: <NEC uPD 9210 USB controller> mem 0x80002000-0x80002fff irq 16 at device 1.0 on pci0
usbus0 on ohci0
ohci1: <NEC uPD 9210 USB controller> mem 0x80001000-0x80001fff irq 17 at device 1.1 on pci0
usbus1 on ohci1
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x80000000-0x800000ff irq 18 at device 1.2 on pci0
usbus2: EHCI version 0.95
usbus2 on ehci0
pci0: <mass storage, ATA> at device 2.0 (no driver attached)
pcib1: <ACPI Host-PCI bridge> on acpi0
pcib1: could not evaluate _ADR - AE_NOT_FOUND
pci32: <ACPI PCI bus> on pcib1
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2100-0x21ff mem 0x903a0000-0x903bffff,0x90380000-0x9039ffff irq 27 at device 1.0 on pci32
mpt0: MPI Version=1.2.12.0
mpt1: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0x90360000-0x9037ffff,0x90340000-0x9035ffff irq 28 at device 1.1 on pci32
mpt1: MPI Version=1.2.12.0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2240-0x227f mem 0x90320000-0x9033ffff,0x90280000-0x902fffff irq 29 at device 2.0 on pci32
em0: Ethernet address: 00:13:21:5b:05:1c
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0x2200-0x223f mem 0x90300000-0x9031ffff irq 30 at device 2.1 on pci32
em1: Ethernet address: 00:13:21:5b:05:1d
pcib2: <ACPI Host-PCI bridge> on acpi0
pcib2: could not evaluate _ADR - AE_NOT_FOUND
pci64: <ACPI PCI bus> on pcib2
pcib3: <ACPI Host-PCI bridge> on acpi0
pcib3: could not evaluate _ADR - AE_NOT_FOUND
pci96: <ACPI PCI bus> on pcib3
pcib4: <ACPI Host-PCI bridge> on acpi0
pcib4: could not evaluate _ADR - AE_NOT_FOUND
pci128: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pcib5: could not evaluate _ADR - AE_NOT_FOUND
pci192: <ACPI PCI bus> on pcib5
isp0: <Qlogic ISP 2422 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xe0040000-0xe0040fff irq 71 at device 1.0 on pci192
pcib6: <ACPI Host-PCI bridge> on acpi0
pcib6: could not evaluate _ADR - AE_NOT_FOUND
pci224: <ACPI PCI bus> on pcib6
uart0: <HP Auxiliary Diva Serial Port> mem 0xf8051000-0xf805100f irq 82 at device 1.0 on pci224
puc0: <HP Diva Serial [GSP] Multiport UART - Everest SP2> mem 0xf8050000-0xf8050fff,0xf8020000-0xf803ffff irq 82 at device 1.1 on pci224
uart1: <Non-standard ns8250 class UART with FIFOs> at port 1 on puc0
uart1: console (9600,n,8,1)
uart2: <Non-standard ns8250 class UART with FIFOs> at port 3 on puc0
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf0000000-0xf7ffffff,0xf8040000-0xf804ffff at device 2.0 on pci224
uart3: <16550 or compatible> iomem 0xff5e0000-0xff5e0007 irq 34 on acpi0
uart4: <16550 or compatible> iomem 0xff5e2000-0xff5e2007 irq 35 on acpi0
uart4: debug port (9600,n,8,1)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
IP Filter: v5.1.2 initialized.  Default = block all, Logging = enabled
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen2.1: <NEC> at usbus2
uhub0: <NEC EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen1.1: <NEC> at usbus1
uhub1: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <NEC> at usbus0
uhub2: <NEC OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub1: 2 ports with 2 removable, self powered
uhub2: 3 ports with 3 removable, self powered
uhub0: 5 ports with 5 removable, self powered
Expensive timeout(9) function: 0x9ffc000000e1e1e0(0x9ffc000000ee3c00) 0.020669666 s
Expensive timeout(9) function: 0x9ffc000000e222e0(0xa0000000009ed320) 0.282418016 s
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <HP 73.4G ST373454LC HPC2> Fixed Direct Access SCSI-3 device
da0: 320.000MB/s transfersWARNING: WITNESS option enabled, expect reduced performance.
 (160.000MHz, offset 63WARNING: DIAGNOSTIC option enabled, expect reduced performance.
, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST318452LC 2213> Fixed Direct Access SCSI-3 device
da1: 160.000MB/s transfers (80.000MHz, offset 63, 16bit)
da1: Command Queueing enabled
da1: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)
da2 at mpt1 bus 0 scbus1 target 2 lun 0
da2: <SEAGATE ST318452LC 2213> Fixed Direct Access SCSI-3 device
da2: 160.000MB/s transfers (80.000MHz, offset 63, 16bit)
da2: Command Queueing enabled
da2: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)
Trying to mount root from ufs:/dev/da0p2 [rw]...
WARNING: / was not properly dismounted
WITNESS: unable to allocate a new witness object
<118>Setting hostuuid: ee652761-3dcc-11df-9074-0013215b051c.
<118>Setting hostid: 0x8968724b.
<118>dumpon: /dev/da4p1: No such file or directory
<118>/etc/rc: WARNING: unable to specify /dev/da4p1 as a dump device
<118>Starting ddb.
<118>Entropy harvesting: interrupts ethernet point_to_point kickstart.
<118>swapon: /dev/da4p1: No such file or directory
<118>Starting file system checks:
da3 at isp0 bus 0 scbus2 target 0 lun 1
da3: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da3: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da3: Command Queueing enabled
da3: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da4 at isp0 bus 0 scbus2 target 0 lun 2
da4: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da4: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da4: Command Queueing enabled
da4: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da5 at isp0 bus 0 scbus2 target 0 lun 3
da5: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da5: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da5: Command Queueing enabled
da5: 69460MB (142255575 512 byte sectors: 255H 63S/T 8855C)
da6 at isp0 bus 0 scbus2 target 0 lun 5
da6: <COMPAQ MSA1000 VOLUME 4.32> Fixed Direct Access SCSI-4 device
da6: 200.000MB/s transfers WWNN 0x500805f3000ec220 WWPN 0x500805f3000ec221 PortID 0x10000
da6: Command Queueing enabled
da6: 140011MB (286744185 512 byte sectors: 255H 63S/T 17849C)
<118>/dev/da0p2: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da0p2: 582356 files, 3927896 used, 30688528 free (183320 frags, 3813151 blocks, 0.5% fragmentation)
<118>/dev/da3p1: INCORRECT BLOCK COUNT I=642145 (8 should be 0) (CORRECTED)
<118>/dev/da3p1: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
<118>/dev/da3p1: SUMMARY INFORMATION BAD (SALVAGED)
<118>/dev/da3p1: BLK(S) MISSING IN BIT MAPS (SALVAGED)
<118>/dev/da3p1: 1967 files, 7223674 used, 9999687 free (263 frags, 1249928 blocks, 0.0% fragmentation)
<118>/dev/da5p1: 436860 files, 1247548 used, 15975813 free (36405 frags, 1992426 blocks, 0.2% fragmentation)
<118>/dev/da6p1: 5481 files, 1323234 used, 68104269 free (5773 frags, 8512312 blocks, 0.0% fragmentation)
<118>Mounting local file systems:.
<118>Writing entropy file:.
<118>Setting hostname: mech-as28.men.bris.ac.uk.
<118>Enabling ipfilter.
<118>Starting Network: lo0 em0 em1.
<118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
<118>   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
<118>   inet6 ::1 prefixlen 128
<118>   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
<118>   inet 127.0.0.1 netmask 0xff000000
<118>   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
<118>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>   ether 00:13:21:5b:05:1c
<118>   inet 137.222.187.28 netmask 0xffffff00 broadcast 137.222.187.255
<118>   inet6 fe80::213:21ff:fe5b:51c%em0 prefixlen 64 scopeid 0x1
<118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>   media: Ethernet autoselect
<118>   status: no carrier
<118>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
<118>   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
<118>   ether 00:13:21:5b:05:1d
<118>   inet 10.10.10.14 netmask 0xffffff00 broadcast 10.10.10.255
<118>   inet6 fe80::213:21ff:fe5b:51d%em1 prefixlen 64 scopeid 0x2
<118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<118>   media: Ethernet autoselect (100baseTX <full-duplex>)
<118>   status: active
<118>filter sync'd
<118>Starting devd.
<118>add net default: gateway 137.222.187.250
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
<118>Starting ipmon.
<118>Creating and/or trimming log files.
<118>Starting syslogd.
<118>No core dumps found.
<118>Clearing /tmp (X related).
<118>Recovering vi editor sessions:.
<118>Updating motd:.
<118>Mounting late file systems:.
<118>Starting ntpd.
<118>Starting bsdstats.
<118>Posting monthly OS statistics to rpt.bsdstats.org
<118>Performing sanity check on sshd configuration.
<118>Starting sshd.
<118>Starting cron.
<118>Local package initialization:cd: /var/portbuild/: No such file or directory
<118>.  
<118>Starting inetd.
<118>Starting background file system checks in 60 seconds.
<118>   
<118>Tue Oct 15 16:15:03 BST 2013
<118>Oct 15 16:15:43 mech-as28 su: mexas to root on /dev/pts/0
<118>Oct 15 16:26:25 mech-as28 su: mexas to root on /dev/pts/1
<118>Oct 15 20:42:07 mech-as28 su: mexas to root on /dev/pts/2
        
db>     

db> show proc
Process 0 (kernel) at 0x9ffc000000efeac8:
 state: NORMAL
 uid: 0  gids: 0
 ABI: null
 threads: 13
100053                   Run     CPU 1                       [deadlkres]
100050                   D       -        0xe000000011795300 [system_taskq_1]
100049                   D       -        0xe000000011795300 [system_taskq_0]
100045                   D       -        0xe0000000111a2500 [em1 taskq]
100044                   D       -        0xe0000000111a2600 [em0 taskq]
100023                   D       -        0xe0000000111a5300 [ffs_trim taskq]
100020                   RunQ                                [thread taskq]
100017                   D       -        0xe0000000111a5900 [acpi_task_2]
100016                   D       -        0xe0000000111a5900 [acpi_task_1]
100015                   D       -        0xe0000000111a5900 [acpi_task_0]
100014                   D       -        0xe0000000111a5a00 [kqueue taskq]
100009                   D       -        0xe0000000111a5d00 [firmware taskq]
100000                   D       swapin   0x9ffc000000efeac8 [swapper]
db> 

db> run lockinfo
db:0:lockinfo> show locks
db:0:locks>  show alllocks
db:0:alllocks>  show lockedvnods
Locked vnodes

0xe0000000120e1ad0: tag devfs, type VDIR
    usecount 3, writecount 0, refcount 4 mountedhere 0
    flags (VV_ROOT|VI_ACTIVE)
    lock type devfs: EXCL by thread 0xe000000015800900 (pid 20716, sh, tid 100129)
 with exclusive waiters pending

0xe0000000120e3860: tag devfs, type VCHR
    usecount 29, writecount 16, refcount 30 mountedhere 0xe0000000111a7800
    flags (VV_ETERNALDEV|VI_ACTIVE)
    lock type devfs: EXCL by thread 0xe0000000157a4d80 (pid 69993, sh, tid 100125)
 with exclusive waiters pending
        dev null
db> 

db> show pcpu
cpuid        = 1
dynamic pcpu = 0x400009f8d2680
curthread    = 0xe00000001178cd80: pid 0 "deadlkres"
curpcb       = 0
fpcurthread  = none
idlethread   = 0xe000000011182480: tid 100004 "idle: cpu1"
MD: vhpt       = 0xe000000011200000
MD: lid        = 0x1000000
MD: clock      = 0x1caaca1b73b6
MD: clock_mode = 2
MD: clock_load = 0
MD: stats      = 0xe000000011158a30
MD: pmap       = 0x9ffc000000eff500
spin locks held:
db> 

db> bt
Tracing pid 0 tid 100053 td 0xe00000001178cd80
db>

db> show threads
  100138 (0xe0000000157e5b00) (stack 0xa0000000002ca000)  ***
  100139 (0xe0000000157e5680) (stack 0xa0000000002d2000)  ***
  100147 (0xe0000001d2c36000) (stack 0xa000000000302000)  ***
  100093 (0xe00000001258b680) (stack 0xa00000009df5e000)  ***
  100120 (0xe00000001574bb00) (stack 0xa00000000023a000)  ***
  100078 (0xe00000001258a000) (stack 0xa00000009defe000)  ***
  100112 (0xe00000001574a000) (stack 0xa00000009dff6000)  ***
  100104 (0xe0000000128ecd80) (stack 0xa00000009dfb6000)  ***
  100151 (0xe0000001d2c37b00) (stack 0xa000000000322000)  ***
  100116 (0xe000000015770000) (stack 0xa00000000021a000)  ***
  100119 (0xe000000015771680) (stack 0xa000000000232000)  ***
  100107 (0xe000000012919b00) (stack 0xa00000009dfce000)  ***
  100117 (0xe0000000157a4000) (stack 0xa000000000222000)  ***
  100126 (0xe0000000157a4900) (stack 0xa00000000026a000)  ***
  100129 (0xe000000015800900) (stack 0xa000000000282000)  ***
  100100 (0xe00000001279bb00) (stack 0xa00000009df96000)  ***
  100125 (0xe0000000157a4d80) (stack 0xa000000000262000)  ***
  100131 (0xe0000000157e4480) (stack 0xa000000000292000)  ***
  100118 (0xe000000015771b00) (stack 0xa00000000022a000)  ***
  100103 (0xe0000000128ed200) (stack 0xa00000009dfae000)  ***
  100056 (0xe000000012168d80) (stack 0xa00000009de4e000)  ***
  100070 (0xe0000000124f8900) (stack 0xa00000009debe000)  ***
  100073 (0xe00000001258a900) (stack 0xa00000009ded6000)  ***
  100098 (0xe0000000128ec480) (stack 0xa00000009df86000)  ***
  100096 (0xe000000012918480) (stack 0xa00000009df76000)  ***
  100099 (0xe0000000128ec000) (stack 0xa00000009df8e000)  ***
  100091 (0xe00000001258b200) (stack 0xa00000009df4e000)  ***
  100065 (0xe000000012168480) (stack 0xa00000009de96000)  ***
  100095 (0xe000000012918900) (stack 0xa00000009df6e000)  ***
  100081 (0xe00000001279a000) (stack 0xa00000009df16000)  ***
  100088 (0xe00000001279ad80) (stack 0xa00000009df36000)  ***
  100068 (0xe0000000124f9200) (stack 0xa00000009deae000)  ***
  100079 (0xe0000000124f9b00) (stack 0xa00000009df06000)  ***
  100076 (0xe0000000123fa900) (stack 0xa00000009deee000)  ***
  100074 (0xe0000000123fb200) (stack 0xa00000009dede000)  ***
  100066 (0xe000000012168000) (stack 0xa00000009de9e000)  ***
  100064 (0xe0000000123fb680) (stack 0xa00000009de8e000)  ***
  100063 (0xe0000000123fbb00) (stack 0xa00000009de86000)  ***
  100062 (0xe000000012169200) (stack 0xa00000009de7e000)  ***
  100061 (0xe000000012169680) (stack 0xa00000009de76000)  ***
  100060 (0xe000000012169b00) (stack 0xa00000009de6e000)  ***
  100059 (0xe0000000123fa000) (stack 0xa00000009de66000)  ***
  100058 (0xe0000000123fa480) (stack 0xa00000009de5e000)  ***
  100057 (0xe000000012168900) (stack 0xa00000009de56000)  ***
  100054 (0xe00000001178c900) (stack 0xa00000009de3e000)  ***
  100052 (0xe00000001178d200) (stack 0xa00000009de2c000)  ***
  100051 (0xe00000001178d680) (stack 0xa00000009de24000)  ***
  100047 (0xe00000001174db00) (stack 0xa00000009de04000)  ***
  100043 (0xe00000001174c000) (stack 0xa00000009ddc6000)  ***
  100041 (0xe00000001174c900) (stack 0xa00000009dd68000)  ***
  100039 (0xe000000011744480) (stack 0xa00000009dd0a000)  ***
  100038 (0xe000000011744900) (stack 0xa00000009dd02000)  ***
  100037 (0xe000000011744d80) (stack 0xa00000009dcfa000)  ***
  100036 (0xe000000011745200) (stack 0xa00000009dcf2000)  ***
  100034 (0xe000000011480d80) (stack 0xa0000000e2fe6000)  ***
  100033 (0xe000000011481200) (stack 0xa0000000e2fde000)  ***
  100032 (0xe000000011481680) (stack 0xa0000000e2fd6000)  ***
  100031 (0xe000000011481b00) (stack 0xa0000000e2fce000)  ***
  100029 (0xe000000011425680) (stack 0xa0000000e2f36000)  ***
  100028 (0xe000000011425b00) (stack 0xa0000000e2f2e000)  ***
  100027 (0xe000000011480000) (stack 0xa0000000e2f26000)  ***
  100026 (0xe000000011480480) (stack 0xa0000000e2f1e000)  ***
  100024 (0xe000000011424000) (stack 0xa0000000e2e8a000)  ***
  100013 (0xe0000000111a1680) (stack 0xa0000000a07f6000)  ***
  100012 (0xe0000000111a1b00) (stack 0xa0000000a07ee000)  ***
  100011 (0xe0000000111b0000) (stack 0xa0000000a07e6000)  ***
  100010 (0xe000000011183680) (stack 0xa0000000a07de000)  ***
  100048 (0xe00000001174d680) (stack 0xa00000009de0c000)  ***
  100046 (0xe00000001178c000) (stack 0xa00000009ddde000)  ***
  100042 (0xe00000001174c480) (stack 0xa00000009dd70000)  ***
  100040 (0xe000000011744000) (stack 0xa00000009dd12000)  ***
  100035 (0xe000000011480900) (stack 0xa00000009dcea000)  ***
  100030 (0xe000000011425200) (stack 0xa0000000e2fc6000)  ***
  100025 (0xe0000000111b1b00) (stack 0xa0000000e2f16000)  ***
  100022 (0xe000000011424900) (stack 0xa0000000e2e7a000)  ***
  100021 (0xe000000011424d80) (stack 0xa0000000e2e72000)  ***
  100019 (0xe0000000111b0900) (stack 0xa0000000e2e62000)  ***
  100018 (0xe0000000111b0d80) (stack 0xa0000000e2e5a000)  ***
  100008 (0xe0000000111a0000) (stack 0xa0000000a07ce000)  ***
  100007 (0xe0000000111a0480) (stack 0xa0000000a07c6000)  ***
  100006 (0xe0000000111a0900) (stack 0xa0000000a07be000)  ***
  100005 (0xe000000011182000) (stack 0xa0000000a07b6000)  ***
  100004 (0xe000000011182480) (stack 0xa0000000a07ae000)  ***
  100003 (0xe000000011182900) (stack 0xa0000000a07a6000)  ***
  100002 (0xe000000011182d80) (stack 0xa0000000a079e000)  ***
  100001 (0xe000000011183200) (stack 0xa0000000a0796000)  ***
  100053 (0xe00000001178cd80) (stack 0xa00000009de34000)  ***
  100050 (0xe00000001174cd80) (stack 0xa00000009de1c000)  ***
  100049 (0xe00000001174d200) (stack 0xa00000009de14000)  ***
  100045 (0xe000000011745680) (stack 0xa00000009ddd6000)  ***
  100044 (0xe000000011745b00) (stack 0xa00000009ddce000)  ***
  100023 (0xe000000011424480) (stack 0xa0000000e2e82000)  ***
  100020 (0xe0000000111b0480) (stack 0xa0000000e2e6a000)  ***
  100017 (0xe0000000111b1200) (stack 0xa0000000e2e52000)  ***
  100016 (0xe0000000111b1680) (stack 0xa0000000e2e4a000)  ***
  100015 (0xe0000000111a0d80) (stack 0xa0000000e2e42000)  ***
  100014 (0xe0000000111a1200) (stack 0xa0000000e2e3a000)  ***
  100009 (0xe000000011183b00) (stack 0xa0000000a07d6000)  ***
  100000 (0x9ffc000000efef70) (stack 0xe0000040fffc0000)  ***
db>


db> ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
37126  1147  1147     0  D+      allproc  0x9ffc000000f0a350 sh
37125 42826  1147     0  DE+     allproc  0x9ffc000000f0a350 sed
37124 42826  1147     0  DE+     allproc  0x9ffc000000f0a350 awk
37123 42826  1147     0  DE+     proctree 0x9ffc000000f0a370 awk
37115 37110  1147     0  S+J     piperd   0xe0000000129509a0 as
37111 37110  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 cc1plus
37110 28849  1147     0  S+J     wait     0xe000000015790940 g++
36890 36889  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 ld
36889 11278  1147     0  SW+J    wait     0xe0000000157664a0 c++
36591 36590  1147     0  LL+J   *pmap pv  0xe00000001216b8c0 cc1plus
36590 14781  1147     0  SW+J    wait     0xe00000001290d280 xg++
28849 61132  1147     0  S+J     wait     0xe000000015767720 gmake
18342 18341 18342  1001  Ds+     allproc  0x9ffc000000f0a350 tcsh
18341 18317 18317  1001  S       select   0xe000000010c286c0 sshd
18317   924 18317     0  Ss      select   0xe00000001273f640 sshd
14781  2824  1147     0  SW+J    wait     0xe00000001572e000 gmake
11278 59577  1147     0  SW+J    wait     0xe000000012762de0 gmake
61132 61124  1147     0  SW+J    wait     0xe0000000157ec4a0 gmake
61124 61123  1147     0  SW+J    wait     0xe000000015791720 gmake
61123 60934  1147     0  SW+J    wait     0xe0000001d4970000 sh
60934 60932  1147     0  SW+J    wait     0xe000000011797280 make
60932 20716  1147     0  SW+     wait     0xe0000001d300cde0 sh
59577 59537  1147     0  SW+J    wait     0xe0000000124a6940 gmake
59537 59516  1147     0  SW+J    wait     0xe000000015791280 gmake
59516 59515  1147     0  SW+J    wait     0xe000000015a1b720 gmake
59515 59505  1147     0  SW+J    wait     0xe00000001572f280 sh
59505 59479  1147     0  SW+J    wait     0xe000000015a1b280 make
59479 59477  1147     0  SW+J    wait     0xe0000000157b9280 make
59477 69993  1147     0  SW+     wait     0xe00000001572e4a0 sh
 2824 77085  1147     0  SW+J    wait     0xe000000015766940 gmake
28012 20716  1147     0  S+      piperd   0xe00000001586a738 tee
20716  1147  1147     0  D+      devfs    0xe0000000120e38c8 sh
77085 77056  1147     0  SW+J    wait     0xe0000000157ecde0 gmake
77056 77055  1147     0  SW+J    wait     0xe000000012762000 gmake
77055 77043  1147     0  SW+J    wait     0xe0000000127ae940 sh
77043 77023  1147     0  SW+J    wait     0xe0000000157b8000 make
77023 77021  1147     0  SW+J    wait     0xe0000000157904a0 make
77021 62311  1147     0  SW+     wait     0xe0000000123f4940 sh
70398 69993  1147     0  S+      piperd   0xe000000012863ce0 tee
69993  1147  1147     0  D+      proctree 0x9ffc000000f0a370 sh
67686 62311  1147     0  S+      piperd   0xe0000001d4ed1340 tee
62311  1147  1147     0  D+      devfs    0xe0000000120e1b38 sh
42826  1147  1147     0  S+      wait     0xe00000001290c940 sh
 1147  1030  1147     0  S+      piperd   0xe00000001586ae70 sh
 1122  1121  1122     0  S+      ttyin    0xe00000001254e4a8 sh
 1121  1047  1121  1001  SW+     wait     0xe00000001289c4a0 su
 1047  1046  1047  1001  SWs+    pause    0xe0000000127ae548 tcsh
 1046  1042  1042  1001  S       select   0xe00000001273fd40 sshd
 1042   924  1042     0  Ss      select   0xe000000010c287c0 sshd
 1030  1029  1030     0  SW+     wait     0xe00000001289d720 sh
 1029  1025  1029  1001  SW+     wait     0xe00000001289c940 su
 1025  1024  1025  1001  SWs+    pause    0xe00000001290c0a8 tcsh
 1024  1022  1022  1001  S       select   0xe000000010c289c0 sshd
 1022   924  1022     0  Ss      select   0xe00000001273f7c0 sshd
  997     1   997     0  Ss+     ttyin    0xe0000000114288a8 getty
  996     1     1     0  S       ttydcd   0xe000000011428ce8 getty
  974     1   974     0  Ss      select   0xe000000010c288c0 inetd
  935     1   935     0  LLs    *pmap pv  0xe00000001216b8c0 cron
  930     1   930    25  SWs     pause    0xe0000000124a7328 sendmail
  927     1   927     0  Rs                                  sendmail
  924     1   924     0  Ss      select   0xe000000010c28e40 sshd
  864     1   864     0  LLs    *pmap pv  0xe00000001216b8c0 ntpd
  784     1   784     0  Rs                                  syslogd
  775     1   775     0  Ss      select   0xe000000010c29040 ipmon
  653     1   653     0  Ss      select   0xe00000001273fe40 devd
  138     1   138     0  SWs     pause    0xe0000000111d77c8 adjkerntz
   23     0     0     0  DL      sdflush  0x9ffc000000ee2d30 [softdepflush]
   22     0     0     0  RL                                  [vnlru]
   21     0     0     0  RL                                  [syncer]
   20     0     0     0  RL                                  [bufdaemon]
   19     0     0     0  DL      pgzero   0x9ffc000000ee30d4 [pagezero]
   18     0     0     0  RL      CPU 0                       [vmdaemon]
   17     0     0     0  LL     *pmap pv  0xe00000001216b8c0 [pagedaemon]
    9     0     0     0  DL      ccb_scan 0x9ffc000000ee33a0 [xpt_thrd]
    6     0     0     0  DL      waiting_ 0x9ffc000001104130 [sctp_iterator]
    5     0     0     0  RL      (threaded)                  [zfskern]
100052                   RunQ                                [l2arc_feed_thread]
100051                   RunQ                                [arc_reclaim_thread]
    4     0     0     0  DL      ispf     0xe000000011768000 [isp0: fc_thrd0]
    3     0     0     0  DL      idle     0xa000000000a16000 [mpt_recovery1]
    2     0     0     0  DL      idle     0xa0000000009fe000 [mpt_recovery0]
   16     0     0     0  DL      (threaded)                  [usb]
100039                   D       -        0xa0000000009fae18 [usbus2]
100038                   D       -        0xa0000000009fadc0 [usbus2]
100037                   D       -        0xa0000000009fad68 [usbus2]
100036                   D       -        0xa0000000009fad10 [usbus2]
100034                   D       -        0xa0000000009f1460 [usbus1]
100033                   D       -        0xa0000000009f1408 [usbus1]
100032                   D       -        0xa0000000009f13b0 [usbus1]
100031                   D       -        0xa0000000009f1358 [usbus1]
100029                   D       -        0xa0000000009ed460 [usbus0]
100028                   D       -        0xa0000000009ed408 [usbus0]
100027                   D       -        0xa0000000009ed3b0 [usbus0]
100026                   D       -        0xa0000000009ed358 [usbus0]
   15     0     0     0  DL      tzpoll   0x9ffc000000ee0a50 [acpi_thermal]
   14     0     0     0  RL                                  [rand_harvestq]
   13     0     0     0  DL      (threaded)                  [geom]
100012                   D       -        0x9ffc000000ee1270 [g_down]
100011                   D       -        0x9ffc000000ee1268 [g_up]
100010                   D       -        0x9ffc000000ee1258 [g_event]
   12     0     0     0  WL      (threaded)                  [intr]
100048                   I                                   [swi0: uart uart+++]
100046                   I                                   [irq71: isp0]
100042                   I                                   [irq28: mpt1]
100040                   I                                   [irq27: mpt0]
100035                   I                                   [irq18: ehci0]
100030                   I                                   [irq17: ohci1]
100025                   I                                   [irq16: ohci0]
100022                   I                                   [swi6: task queue]
100021                   I                                   [swi6: Giant taskq]
100019                   I                                   [swi5: fast taskq]
100018                   I                                   [swi2: cambio]
100008                   I                                   [swi4: clock]
100007                   I                                   [swi4: clock]
100006                   I                                   [swi3: vm]
100005                   I                                   [swi1: netisr 0]
   11     0     0     0  RL      (threaded)                  [idle]
100004                   CanRun                              [idle: cpu1]
100003                   CanRun                              [idle: cpu0]
    1     0     1     0  SLs     wait     0xe000000011180de0 [init]
   10     0     0     0  DL      audit_wo 0x9ffc0000011087c8 [audit]
    0     0     0     0  RLs     (threaded)                  [kernel]
100053                   Run     CPU 1                       [deadlkres]
100050                   D       -        0xe000000011795300 [system_taskq_1]
100049                   D       -        0xe000000011795300 [system_taskq_0]
100045                   D       -        0xe0000000111a2500 [em1 taskq]
100044                   D       -        0xe0000000111a2600 [em0 taskq]
100023                   D       -        0xe0000000111a5300 [ffs_trim taskq]
100020                   RunQ                                [thread taskq]
100017                   D       -        0xe0000000111a5900 [acpi_task_2]
100016                   D       -        0xe0000000111a5900 [acpi_task_1]
100015                   D       -        0xe0000000111a5900 [acpi_task_0]
100014                   D       -        0xe0000000111a5a00 [kqueue taskq]
100009                   D       -        0xe0000000111a5d00 [firmware taskq]
100000                   D       swapin   0x9ffc000000efeac8 [swapper]
db> 

db> alltrace

Tracing command sh pid 37126 tid 100138 td 0xe0000000157e5b00

Tracing command sed pid 37125 tid 100139 td 0xe0000000157e5680

Tracing command awk pid 37124 tid 100147 td 0xe0000001d2c36000

Tracing command awk pid 37123 tid 100093 td 0xe00000001258b680

Tracing command as pid 37115 tid 100120 td 0xe00000001574bb00

Tracing command cc1plus pid 37111 tid 100078 td 0xe00000001258a000

Tracing command g++ pid 37110 tid 100112 td 0xe00000001574a000

Tracing command ld pid 36890 tid 100104 td 0xe0000000128ecd80

Tracing command c++ pid 36889 tid 100115 td 0xe000000012919200

Tracing command cc1plus pid 36591 tid 100151 td 0xe0000001d2c37b00
        
Tracing command xg++ pid 36590 tid 100101 td 0xe0000000128edb00

Tracing command gmake pid 28849 tid 100116 td 0xe000000015770000

Tracing command tcsh pid 18342 tid 100119 td 0xe000000015771680

Tracing command sshd pid 18341 tid 100107 td 0xe000000012919b00

Tracing command sshd pid 18317 tid 100117 td 0xe0000000157a4000

Tracing command gmake pid 14781 tid 100111 td 0xe00000001574a480

Tracing command gmake pid 11278 tid 100077 td 0xe00000001258a480

Tracing command gmake pid 61132 tid 100134 td 0xe000000015801680

Tracing command gmake pid 61124 tid 100122 td 0xe000000015770d80

Tracing command sh pid 61123 tid 100149 td 0xe0000001d492c480
        
Tracing command make pid 60934 tid 100067 td 0xe00000001178db00

Tracing command sh pid 60932 tid 100141 td 0xe000000015801200

Tracing command gmake pid 59577 tid 100071 td 0xe0000000124f8480

Tracing command gmake pid 59537 tid 100123 td 0xe0000000157a5680

Tracing command gmake pid 59516 tid 100145 td 0xe0000001d2c36480

Tracing command sh pid 59515 tid 100105 td 0xe0000000128ec900

Tracing command make pid 59505 tid 100135 td 0xe000000015800480

Tracing command make pid 59479 tid 100127 td 0xe0000000157a4480

Tracing command sh pid 59477 tid 100110 td 0xe00000001574a900

Tracing command gmake pid 2824 tid 100109 td 0xe000000012919680
        
Tracing command tee pid 28012 tid 100126 td 0xe0000000157a4900

Tracing command sh pid 20716 tid 100129 td 0xe000000015800900

Tracing command gmake pid 77085 tid 100132 td 0xe0000000157e4000

Tracing command gmake pid 77056 tid 100080 td 0xe0000000124f9680

Tracing command sh pid 77055 tid 100085 td 0xe00000001258ad80

Tracing command make pid 77043 tid 100121 td 0xe000000015771200

Tracing command make pid 77023 tid 100113 td 0xe000000015770900

Tracing command sh pid 77021 tid 100075 td 0xe0000000123fad80

Tracing command tee pid 70398 tid 100100 td 0xe00000001279bb00

Tracing command sh pid 69993 tid 100125 td 0xe0000000157a4d80
        
Tracing command tee pid 67686 tid 100131 td 0xe0000000157e4480

Tracing command sh pid 62311 tid 100118 td 0xe000000015771b00

Tracing command sh pid 42826 tid 100103 td 0xe0000000128ed200

Tracing command sh pid 1147 tid 100056 td 0xe000000012168d80

Tracing command sh pid 1122 tid 100070 td 0xe0000000124f8900

Tracing command su pid 1121 tid 100090 td 0xe00000001258bb00

Tracing command tcsh pid 1047 tid 100086 td 0xe00000001279b680

Tracing command sshd pid 1046 tid 100073 td 0xe00000001258a900

Tracing command sshd pid 1042 tid 100098 td 0xe0000000128ec480

Tracing command sh pid 1030 tid 100094 td 0xe000000012918d80
        
Tracing command su pid 1029 tid 100097 td 0xe000000012918000
        
Tracing command tcsh pid 1025 tid 100092 td 0xe00000001279a480
        
Tracing command sshd pid 1024 tid 100096 td 0xe000000012918480

Tracing command sshd pid 1022 tid 100099 td 0xe0000000128ec000

Tracing command getty pid 997 tid 100091 td 0xe00000001258b200

Tracing command getty pid 996 tid 100065 td 0xe000000012168480

Tracing command inetd pid 974 tid 100095 td 0xe000000012918900

Tracing command cron pid 935 tid 100081 td 0xe00000001279a000

Tracing command sendmail pid 930 tid 100069 td 0xe0000000124f8d80

Tracing command sendmail pid 927 tid 100088 td 0xe00000001279ad80

Tracing command sshd pid 924 tid 100068 td 0xe0000000124f9200

Tracing command ntpd pid 864 tid 100079 td 0xe0000000124f9b00

Tracing command syslogd pid 784 tid 100076 td 0xe0000000123fa900

Tracing command ipmon pid 775 tid 100074 td 0xe0000000123fb200

Tracing command devd pid 653 tid 100066 td 0xe000000012168000

Tracing command adjkerntz pid 138 tid 100055 td 0xe00000001178c480

Tracing command softdepflush pid 23 tid 100064 td 0xe0000000123fb680

Tracing command vnlru pid 22 tid 100063 td 0xe0000000123fbb00

Tracing command syncer pid 21 tid 100062 td 0xe000000012169200

Tracing command bufdaemon pid 20 tid 100061 td 0xe000000012169680

Tracing command pagezero pid 19 tid 100060 td 0xe000000012169b00

Tracing command vmdaemon pid 18 tid 100059 td 0xe0000000123fa000

Tracing command pagedaemon pid 17 tid 100058 td 0xe0000000123fa480

Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012168900

Tracing command sctp_iterator pid 6 tid 100054 td 0xe00000001178c900

Tracing command zfskern pid 5 tid 100052 td 0xe00000001178d200

Tracing command zfskern pid 5 tid 100051 td 0xe00000001178d680

Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe00000001174db00

Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe00000001174c000

Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe00000001174c900

Tracing command usb pid 16 tid 100039 td 0xe000000011744480

Tracing command usb pid 16 tid 100038 td 0xe000000011744900

Tracing command usb pid 16 tid 100037 td 0xe000000011744d80

Tracing command usb pid 16 tid 100036 td 0xe000000011745200

Tracing command usb pid 16 tid 100034 td 0xe000000011480d80

Tracing command usb pid 16 tid 100033 td 0xe000000011481200

Tracing command usb pid 16 tid 100032 td 0xe000000011481680

Tracing command usb pid 16 tid 100031 td 0xe000000011481b00

Tracing command usb pid 16 tid 100029 td 0xe000000011425680

Tracing command usb pid 16 tid 100028 td 0xe000000011425b00

Tracing command usb pid 16 tid 100027 td 0xe000000011480000

Tracing command usb pid 16 tid 100026 td 0xe000000011480480

Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011424000

Tracing command rand_harvestq pid 14 tid 100013 td 0xe0000000111a1680

Tracing command geom pid 13 tid 100012 td 0xe0000000111a1b00

Tracing command geom pid 13 tid 100011 td 0xe0000000111b0000

Tracing command geom pid 13 tid 100010 td 0xe000000011183680

Tracing command intr pid 12 tid 100048 td 0xe00000001174d680

Tracing command intr pid 12 tid 100046 td 0xe00000001178c000

Tracing command intr pid 12 tid 100042 td 0xe00000001174c480

Tracing command intr pid 12 tid 100040 td 0xe000000011744000

Tracing command intr pid 12 tid 100035 td 0xe000000011480900


Tracing command intr pid 12 tid 100030 td 0xe000000011425200

Tracing command intr pid 12 tid 100025 td 0xe0000000111b1b00

Tracing command intr pid 12 tid 100022 td 0xe000000011424900

Tracing command intr pid 12 tid 100021 td 0xe000000011424d80

Tracing command intr pid 12 tid 100019 td 0xe0000000111b0900

Tracing command intr pid 12 tid 100018 td 0xe0000000111b0d80

Tracing command intr pid 12 tid 100008 td 0xe0000000111a0000

Tracing command intr pid 12 tid 100007 td 0xe0000000111a0480

Tracing command intr pid 12 tid 100006 td 0xe0000000111a0900

Tracing command intr pid 12 tid 100005 td 0xe000000011182000

Tracing command idle pid 11 tid 100004 td 0xe000000011182480

Tracing command idle pid 11 tid 100003 td 0xe000000011182900

Tracing command init pid 1 tid 100002 td 0xe000000011182d80

Tracing command audit pid 10 tid 100001 td 0xe000000011183200

Tracing command kernel pid 0 tid 100053 td 0xe00000001178cd80

Tracing command kernel pid 0 tid 100050 td 0xe00000001174cd80

Tracing command kernel pid 0 tid 100049 td 0xe00000001174d200

Tracing command kernel pid 0 tid 100045 td 0xe000000011745680

Tracing command kernel pid 0 tid 100044 td 0xe000000011745b00

Tracing command kernel pid 0 tid 100023 td 0xe000000011424480

Tracing command kernel pid 0 tid 100020 td 0xe0000000111b0480

Tracing command kernel pid 0 tid 100017 td 0xe0000000111b1200

Tracing command kernel pid 0 tid 100016 td 0xe0000000111b1680

Tracing command kernel pid 0 tid 100015 td 0xe0000000111a0d80

Tracing command kernel pid 0 tid 100014 td 0xe0000000111a1200

Tracing command kernel pid 0 tid 100009 td 0xe000000011183b00

Tracing command kernel pid 0 tid 100000 td 0x9ffc000000efef70
db>  

db> show pginfo 0xe0000000128ecd80  
page 0xe0000000128ecd80 obj 0 pidx 0x9ffc000000f27760 phys 0x0 q 0 hold 0 wire 10074176
  af 0x0 of 0x0 f 0xa000 act 24 busy 1273e880 valid 0x1240 dirty 0x0
db> 

db> textdump dump
db> call doadump
= 0x6
db> 

	
>How-To-Repeat:
	
>Fix:

	


>Release-Note:
>Audit-Trail:
>Unformatted:



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201310152134.r9FLYE06010287>