Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 9 Nov 2011 16:31:29 GMT
From:      Moritz Wilhelmy <mw@wzff.de>
To:        FreeBSD-gnats-submit@FreeBSD.org
Subject:   kern/162407: rctl crashes kernel with a page fault (gdb: "corrupt stack?")
Message-ID:  <201111091631.pA9GVTpF061747@phallus.wzff.de>
Resent-Message-ID: <201111091640.pA9Ge8hY022833@freefall.freebsd.org>

next in thread | raw e-mail | index | archive | help

>Number:         162407
>Category:       kern
>Synopsis:       rctl crashes kernel with a page fault (gdb: "corrupt stack?")
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 09 16:40:08 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Moritz Wilhelmy
>Release:        FreeBSD 9.0-RC2 amd64
>Organization:
>Environment:
System: FreeBSD phallus.wzff.de 9.0-RC2 FreeBSD 9.0-RC2 #14: Wed Nov 9 10:27:26 UTC 2011 root@phallus.wzff.de:/usr/obj/usr/src/sys/PHALLUS amd64

Automatically generated informative text from the first crash (was saved as core.txt.1):

phallus.wzff.de dumped core - see /pool/crash/vmcore.1

Sun Oct 30 22:37:10 UTC 2011

FreeBSD phallus.wzff.de 9.0-RC1 FreeBSD 9.0-RC1 #10: Sun Oct 30 11:34:41 UTC 2011     root@phallus.wzff.de:/usr/obj/usr/src/sys/PHALLUS  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address	= 0x360
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff804c4b9a
stack pointer	        = 0x28:0xffffff845da34780
frame pointer	        = 0x28:0xffffff845da347b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1369 (ntpd)
trap number		= 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff804fb9de at kdb_backtrace+0x5e
#1 0xffffffff804c6537 at panic+0x187
#2 0xffffffff8073a060 at trap_fatal+0x290
#3 0xffffffff8073a3a9 at trap_pfault+0x1f9
#4 0xffffffff8073a86f at trap+0x3df
#5 0xffffffff80724d9f at calltrap+0x8
#6 0xffffffff806530aa at in6_setscope+0x23a
#7 0xffffffff8063bdbb at in6_selectsrc+0x36b
#8 0xffffffff80639309 at in6_pcbladdr+0x149
#9 0xffffffff806399c4 at in6_pcbconnect_mbuf+0x44
#10 0xffffffff80656844 at udp6_connect+0x1a4
#11 0xffffffff80536e51 at kern_connect+0x121
#12 0xffffffff80536fc1 at sys_connect+0x41
#13 0xffffffff80739950 at amd64_syscall+0x450
#14 0xffffffff80725087 at Xfast_syscall+0xf7
GEOM_MIRROR: Device zfs: rebuilding provider ad4p7 stopped.
Uptime: 36m34s
Dumping 753 out of 16284 MB:..3%..11%..22%..32%..41%..51%..62%..71%..81%..92%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff804c6075 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:442
#2  0xffffffff804c6521 in panic (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/kern/kern_shutdown.c:607
#3  0xffffffff8073a060 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:818
#4  0xffffffff8073a3a9 in trap_pfault (frame=0xffffff845da346d0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:734
#5  0xffffffff8073a86f in trap (frame=0xffffff845da346d0)
    at /usr/src/sys/amd64/amd64/trap.c:473
#6  0xffffffff80724d9f in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff804c4b9a in _rw_wlock_hard (rw=0xfffffe0011179bc0, 
    tid=18446741874976456704, file=Variable "file" is not available.
) at /usr/src/sys/kern/kern_rwlock.c:677
#8  0xffffffff806530aa in in6_setscope (in6=0xffffff845da348d0, 
    ifp=0xfffffe0011179800, ret_id=0xffffff845da34880)
    at /usr/src/sys/netinet6/scope6.c:417
#9  0xffffffff8063bdbb in in6_selectsrc (dstsock=0xfffffe0004e8c840, 
    opts=0x0, inp=0xfffffe00114a47a8, ro=Variable "ro" is not available.
)
    at /usr/src/sys/netinet6/in6_src.c:309
#10 0xffffffff80639309 in in6_pcbladdr (inp=0xfffffe00114a47a8, 
    nam=0xfffffe0004e8c840, plocal_addr6=0xffffff845da349e0)
    at /usr/src/sys/netinet6/in6_pcb.c:332
#11 0xffffffff806399c4 in in6_pcbconnect_mbuf (inp=0xfffffe00114a47a8, 
    nam=0xfffffe0004e8c840, cred=0xfffffe0004318d00, m=0x0)
    at /usr/src/sys/netinet6/in6_pcb.c:383
#12 0xffffffff80656844 in udp6_connect (so=0xfffffe00114a2000, 
    nam=0xfffffe0004e8c840, td=0xfffffe00114b8000)
    at /usr/src/sys/netinet6/udp6_usrreq.c:1000
#13 0xffffffff80536e51 in kern_connect (td=0xfffffe00114b8000, fd=Variable "fd" is not available.
)
    at /usr/src/sys/kern/uipc_syscalls.c:569
#14 0xffffffff80536fc1 in sys_connect (td=0xfffffe00114b8000, 
    uap=0xffffff845da34bc0) at /usr/src/sys/kern/uipc_syscalls.c:534
#15 0xffffffff80739950 in amd64_syscall (td=0xfffffe00114b8000, traced=0)
    at subr_syscall.c:131
#16 0xffffffff80725087 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:387
#17 0x000000080135217c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
ps -axl

  UID   PID  PPID CPU PRI NI    VSZ    RSS MWCHAN STAT  TT       TIME COMMAND
    0     0     0   0  -8  0      0      0 -      DLs   ??    0:00.09 [kernel]
    0     1     0   0  20  0   6280      0 wait   DLs   ??    0:00.00 [init]
    0     2     0   0 -16  0      0      0 pftm   DL    ??    0:00.00 [pfpurge]
    0     3     0   0 -16  0      0      0 waitin DL    ??    0:00.00 [sctp_ite
    0     4     0   0  -8  0      0      0 m:w1   DL    ??    0:00.01 [g_mirror
    0     5     0   0  -8  0      0      0 m:w1   DL    ??    0:00.01 [g_mirror
    0     6     0   0  -8  0      0      0 m:w1   DL    ??    0:00.00 [g_mirror
    0     7     0   0 -16  0      0      0 ccb_sc DL    ??    0:00.00 [xpt_thrd
    0     8     0   0 -16  0      0      0 psleep DL    ??    0:00.00 [pagedaem
    0     9     0   0 -16  0      0      0 psleep DL    ??    0:00.02 [vmdaemon
    0    10     0   0 -16  0      0      0 audit_ DL    ??    0:00.00 [audit]
    0    11     0   0 155  0      0      0 -      RL    ??  240:02.29 [idle]
    0    12     0   0 -72  0      0      0 -      WL    ??    0:22.74 [intr]
    0    13     0   0  -8  0      0      0 -      DL    ??    0:58.45 [geom]
    0    14     0   0 -16  0      0      0 -      DL    ??    0:02.32 [yarrow]
    0    15     0   0 -68  0      0      0 -      DL    ??    0:00.07 [usb]
    0    16     0   0 155  0      0      0 pgzero DL    ??    0:00.00 [pagezero
    0    17     0   0 -16  0      0      0 psleep DL    ??    0:00.00 [bufdaemo
    0    18     0   0  16  0      0      0 -      RL    ??    0:00.02 [syncer]
    0    19     0   0 -16  0      0      0 vlruwt DL    ??    0:00.00 [vnlru]
    0    20     0   0 -16  0      0      0 sdflus DL    ??    0:00.01 [softdepf
    0    21     0   0  16  0      0      0 c_flow DL    ??    0:00.01 [flowclea
    0    22     0   0  -8  0      0      0 m:w1   DL    ??    0:00.00 [g_mirror
    0    23     0   0  -8  0      0      0 m:w1   DL    ??    0:25.24 [g_mirror
    0    24     0   0 -16  0      0      0 -      DL    ??    0:00.04 [racctd]
    0    42     0   0  -8  0      0      0 tx->tx DL    ??    0:00.04 [zfskern]
    0   957     1   0  20  0  10372      0 select Ds    ??    0:00.01 [devd]
    0  1195     1   0   1  0  12184      0 -      Rs    ??    0:00.01 [syslogd]
    0  1369     1   0  20  0  22332      0 -      Rs    ??    0:00.05 [ntpd]
    0  1422     1   0  20  0  46876      0 select Ds    ??    0:00.00 [sshd]
    0  1450     1   0  20  0  20384      0 select Ds    ??    0:00.01 [sendmail
   25  1454     1   0  20  0  20384      0 pause  Ds    ??    0:00.00 [sendmail
    0  1460     1   0  52  0  14260      0 nanslp Ds    ??    0:00.00 [cron]
    0  1915     1   0  52  0  12184      0 ttyin  Ds+   ??    0:00.00 [getty]
    0  3894     1   0  52  0  10052      0 select Ds    ??    0:00.00 [dhclient
   65  3946     1   0  20  0  10052      0 select Ds    ??    0:00.00 [dhclient
    0  3947  1422   0   4  0  68016      0 -      Rs    ??    0:00.05 [sshd]
    0  3949  3947   0  20  0  14612      0 pause  Ds    ??    0:00.01 [csh]
    0  4022  3949   0  21  0  18884      0 select D+    ??    0:00.00 [tmux]
    0  4024     1   0  20  0  18884      0 -      Rs    ??    0:00.14 [tmux]
    0  4025  4024   0  32  0  14612      0 pause  Ds    ??    0:00.01 [csh]
    0  4555  4024   0  21  0  14612      0 pause  Ds    ??    0:00.01 [csh]
    0  4559  4555   0  52  0  14636      0 wait   D+    ??    0:00.00 [sh]
    0  4569  4559   0  20  0  12428      0 ttyin  D+    ??    0:00.01 [less]
    0  5571     1   0  20  0   6924      0 select Ds    ??    0:00.00 [syslogd]
    0  5698     1   0  20  0  12060      0 select Ds    ??    0:00.00 [sendmail
   25  5702     1   0  52  0  12060      0 pause  Ds    ??    0:00.00 [sendmail
    0  5709     1   0  48  0   7980      0 nanslp Ds    ??    0:00.00 [cron]
    0  5736  4025   0  20  0   8288      0 ttyin  D+    ??    0:00.01 [csh]
    0  5768  4024   0  20  0  14612      0 ttyin  Ds+   ??    0:00.01 [csh]

------------------------------------------------------------------------
vmstat -s

 54412709 cpu context switches
  4531311 device interrupts
   119208 software interrupts
   411083 traps
   560308 system calls
       25 kernel threads created
     5638  fork() calls
      110 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     1606 vnode pager pageins
     4219 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
      941 pages reactivated
   175576 copy-on-write faults
      678 copy-on-write optimized faults
   109786 zero fill pages zeroed
        0 zero fill pages prezeroed
       57 intransit blocking page faults
   411765 total VM faults taken
        0 pages affected by kernel thread creation
  2453783 pages affected by  fork()
    46902 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   426505 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     5500 pages active
     3556 pages inactive
       10 pages in VM cache
    58224 pages wired down
  3970361 pages free
     4096 bytes per page
   107081 total name lookups
          cache hits (85% pos + 9% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
         cred    62    10K       -    15792  64,256
         rctl     0     0K       -        8  32,64
      uidinfo     4     3K       -       32  128,2048
       plimit     9     3K       -      469  256
       DEVFS2    95     2K       -       97  16
   DEVFS_RULE    54    26K       -      207  64,512
        DEVFS    27     1K       -       67  16,32,128
    sysctltmp     0     0K       -      850  16,32,64,128,256,512,4096
    sysctloid  4140   204K       -     4247  16,32,64,128
       sysctl     0     0K       -     1965  16,32,64
      tidhash     1    16K       -        1  
      callout     7  3584K       -        7  
         umtx   978   123K       -      978  128
     p1003.1b     1     1K       -        1  16
         SWAP     4  4377K       -        4  64
       DEVFSP     1     1K       -        3  64
       bus-sc    86   140K       -     1431  16,32,64,128,256,512,1024,2048,4096
          bus   879    79K       -     3721  16,32,64,128,256,1024
      devstat    16    33K       -       16  32,4096
 eventhandler    77     7K       -       77  64,128
    pfs_nodes    21     6K       -       21  256
         kobj   174   696K       -      282  4096
      Per-cpu     1     1K       -        1  32
     acpitask     1     2K       -        1  2048
         GEOM   201    34K       -      955  16,32,64,128,256,512,1024
      entropy  1024    64K       -     1024  64
         rman   208    24K       -      458  16,32,128
      acpidev    40     3K       -       40  64
         sbuf     0     0K       -     1532  16,32,64,128,256,512,1024,2048,4096
   CAM periph     2     1K       -        5  16,32,64,256
      CAM XPT     5     3K       -       12  32,64,128,2048
        stack     0     0K       -        2  256
    taskqueue    55     6K       -       85  16,32,64,128,1024
       Unitno    12     1K       -    16068  32,64
          iov     0     0K       -      727  16,64,128,256,512
       select    48     6K       -       48  128
     ioctlops     0     0K       -    10228  16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    21    21K       -       26  1024,2048
          pts     4     1K       -        7  256
     mbuf_tag     1     1K       -     7417  32,64,128
         ksem     1     8K       -        1  
        shmfd     1     8K       -        1  
      acpisem    28     4K       -       28  128
          pcb    18   157K       -      212  16,32,128,1024,2048,4096
       soname     8     1K       -     1515  16,32,128
          acl     0     0K       -        6  4096
     vfscache     1  4096K       -        1  
     vfs_hash     1  2048K       -        1  
  ata_generic     2     2K       -        2  1024
       vnodes     2     1K       -        2  256
  vnodemarker     1     1K       -     1268  512
        mount   114     5K       -      516  16,32,64,128,256,1024
          BPF    11    10K       -       22  128,512,4096
  ether_multi    45     3K       -      148  16,32,64
       ifaddr    90    28K       -      109  32,64,128,256,512,2048,4096
        ifnet     7    13K       -        8  128,2048
        clone     7    28K       -        7  4096
       arpcom     1     1K       -        1  16
      lltable    31    11K       -       39  256,512
      ata_pci     1     1K       -        1  64
    ad_driver     2     1K       -        2  32
    ar_driver     0     0K       -       12  512,2048
       kbdmux     6    18K       -        6  16,512,1024,2048
  mirror_data    15     4K       -    23317  16,64,128,512
     routetbl    70  4128K       -      607  32,64,128,256,512
         igmp     6     2K       -        7  256
         ipid     2    24K       -        2  
       isadev     7     1K       -        7  128
     in_multi     2     1K       -        9  256
    sctp_iter     0     0K       -       32  256
     sctp_ifn     2     1K       -        7  128
     sctp_ifa     9     2K       -       23  128
     sctp_vrf     1     1K       -        2  64
    sctp_a_it     0     0K       -       32  16
    hostcache     1    28K       -        1  
     syncache     1    96K       -        1  
       USBdev    16     5K       -       16  64,128,1024
          USB    32    30K       -       36  16,32,128,2048
    in6_multi    35     4K       -       90  32,256
         cdev     8     2K       -        8  256
          mld     6     1K       -        7  128
      NFS FHA     1     2K       -        1  2048
          rpc     2     9K       -        2  256
audit_evclass   179     6K       -      218  32
     savedino     0     0K       -       96  256
     freework     1     1K       -       84  16,128
    newdirblk     0     0K       -       13  64
       dirrem     0     0K       -      156  128
        mkdir     0     0K       -       26  128
       diradd     0     0K       -      167  128
     freefile     1     1K       -      105  64
     freeblks     0     0K       -       80  128
     freefrag     0     0K       -       11  128
       newblk     1   512K       -      117  256
    bmsafemap     1     8K       -       74  256
     inodedep     9  2052K       -      272  512
      pagedep     1   512K       -       59  256
  ufs_dirhash    39     8K       -       39  16,32,64,512
    ufs_mount    12    50K       -       13  512,1024,4096
    vm_pgdata     3   129K       -        3  128
      UMAHash     1     1K       -        1  512
     pci_link    16     2K       -       16  64,128
        sigio     1     1K       -        1  64
     filedesc    51    26K       -     5775  512,1024
         kenv   109    13K       -      114  16,32,64,128
       kqueue     0     0K       -      154  256,2048
    acpi_perf     8     4K       -        8  512
    proc-args    25     2K       -     2621  16,32,64,128,256
      memdesc     1     4K       -        1  4096
        hhook     2     1K       -        2  128
     atkbddev     1     1K       -        1  64
      ithread    97    17K       -       97  32,128,256
 prison_racct     3     2K       -        3  512
       prison     3     5K       -       10  16,4096
       KTRACE   100    13K       -      100  128
CAM dev queue     1     1K       -        1  128
       linker   112   786K       -      131  16,32,64,128,256,512,2048,4096
        lockf    29     3K       -      133  64,128
   loginclass     2     1K       -       20  64
    CAM queue     3     1K       -        7  16
       ip6ndp    10     1K       -       21  64,128
       apmdev     1     1K       -        1  128
       ip6opt     0     0K       -        8  256
         temp    67     4K       -     8538  16,32,64,128,256,512,1024,2048,4096
       devbuf  3941  8391K       -     4133  16,32,64,128,256,512,1024,2048,4096
      CAM SIM     1     1K       -        1  256
       module   277    35K       -      277  128
     acpiintr     1     1K       -        1  64
     mtx_pool     2    16K       -        2  
       acpica  4480   456K       -    48311  16,32,64,128,256,512,1024,2048,4096
      io_apic     1     2K       -        1  2048
       DEVFS1    96    48K       -      106  512
          osd     3     1K       -        4  16,64
       DEVFS3   236    59K       -      608  256
          MCA     9     2K       -        9  64,128
          msi     9     2K       -        9  128
     nexusdev     3     1K       -        3  16
      subproc   265   308K       -     5989  512,4096
         proc     2    16K       -        2  
      session    21     3K       -       69  128
         pgrp    24     3K       -      137  128
      solaris 10482 56658K       -   148905  16,32,64,128,256,512,1024,2048,4096
   kstat_data     4     1K       -        4  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     206,      15,     206,   0,   0
UMA Zones:             1408,      0,     206,       0,     206,   0,   0
UMA Slabs:              568,      0,    2277,      47,    4862,   0,   0
UMA RCntSlabs:          568,      0,    1143,       5,    1143,   0,   0
UMA Hash:               256,      0,      80,      10,      81,   0,   0
16 Bucket:              152,      0,     208,      17,     208,   0,   0
32 Bucket:              280,      0,     233,       5,     233,   1,   0
64 Bucket:              536,      0,     243,       2,     243,  84,   0
128 Bucket:            1048,      0,     278,       1,     278, 685,   0
VM OBJECT:              216,      0,    1709,     757,   74341,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 529387,      74,     980,    9150,   0,   0
MAP ENTRY:              120,      0,     651,     744,  164462,   0,   0
fakepg:                 120,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     244,       3,     244,   0,   0
16:                      16,      0,    2730,     798,   50648,   0,   0
32:                      32,      0,    2136,    1298,   35335,   0,   0
64:                      64,      0,   12448,    5078,   96754,   0,   0
128:                    128,      0,    6686,     854,   37043,   0,   0
256:                    256,      0,     988,     782,   25202,   0,   0
512:                    512,      0,    1026,     353,   55369,   0,   0
1024:                  1024,      0,     121,     899,    5920,   0,   0
2048:                  2048,      0,     271,     141,     987,   0,   0
4096:                  4096,      0,     410,     281,   12248,   0,   0
Files:                   80,      0,     111,     654,   23974,   0,   0
TURNSTILE:              136,      0,     490,      70,     490,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
rctl_rule_link:          32,      0,      28,     982,    7028,   0,   0
rctl_rule:               80,      0,       8,      82,      16,   0,   0
racct:                  168,      0,      58,     316,    5828,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1160,      0,      49,     164,    5773,   0,   0
THREAD:                1112,      0,     397,      92,     547,   0,   0
SLEEPQUEUE:              80,      0,     490,     148,     490,   0,   0
VMSPACE:                392,      0,      25,     205,    5750,   0,   0
cpuset:                  72,      0,       3,     247,      20,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     257,    1674,    4915,   0,   0
mbuf:                   256,      0,      21,    1015,    7646,   0,   0
mbuf_cluster:          2048,  25600,    1920,     258,    1948,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,      54,      54,   0,   0
mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  232,      0,       0,     432,11409098,   0,   0
ttyinq:                 160,      0,     120,     120,     285,   0,   0
ttyoutq:                256,      0,      63,      72,     149,   0,   0
ata_request:            320,      0,       1,     363, 4556238,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
VNODE:                  480,      0,     994,     214,    1195,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,     128,   43175,   0,   0
S VFS Cache:            108,      0,     958,     263,    3896,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      48,      20,      48,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                648,      0,       0,       0,       0,   0,   0
Mountpoints:            768,      0,      10,      15,      13,   0,   0
pipe:                   728,      0,       3,     182,    3995,   0,   0
ksiginfo:               112,      0,     183,     873,     405,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
KNOTE:                  128,      0,       0,     174,     184,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,       2,       6,       2,   0,   0
pfstatepl:              288,  10010,       8,      44,      56,   0,   0
pfstatekeypl:           288,      0,       5,      60,      89,   0,   0
pfstateitempl:          288,      0,       5,      60,      89,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       1,      83,       1,   0,   0
pfrktable:             1296,   1002,       2,      22,      21,   0,   0
pfrkentry:              160, 200016,       4,     116,      16,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       9,     327,      42,   0,   0
pfiaddrpl:              120,      0,       1,      61,       1,   0,   0
pfospfen:               112,      0,     700,      26,     700,   0,   0
pfosfp:                  40,      0,     410,      94,     410,   0,   0
pfsync:                  88,      0,       0,       0,       0,   0,   0
socket:                 680,  25602,      45,      87,    1234,   0,   0
unpcb:                  240,  25600,      21,     155,     270,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      16,     104,     889,   0,   0
udpcb:                   16,  25704,      16,     992,     889,   0,   0
tcp_inpcb:              392,  25600,       5,      65,      39,   0,   0
tcpcb:                  976,  25600,       5,      31,      39,   0,   0
tcptw:                   72,   5150,       0,       0,       0,   0,   0
syncache:               152,  15375,       0,      50,       5,   0,   0
hostcache:              136,  15372,       3,      81,       3,   0,   0
tcpreass:                40,   1680,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
sctp_ep:               1368,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2280,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     360,      36,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       1,      49,      11,   0,   0
rtentry:                200,      0,      25,      70,      59,   0,   0
selfd:                   56,      0,      82,     485,   79258,   0,   0
SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
ip4flow:                 56, 197694,       2,    1069,     154,   0,   0
ip6flow:                 80, 197640,       0,     675,      49,   0,   0
FFS inode:              168,      0,     613,     179,     718,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,     613,     182,     718,   0,   0
taskq_zone:              48,      0,       0,     504,      30,   0,   0
zio_cache:              880,      0,       1,     903,   35072,   0,   0
zio_link_cache:          48,      0,       0,    1656,   16416,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,     334,     296,     408,   0,   0
dnode_t:                856,      0,     483,      57,     633,   0,   0
dmu_buf_impl_t:         224,      0,     891,     282,    1170,   0,   0
arc_buf_hdr_t:          216,      0,    1053,     153,    1176,   0,   0
arc_buf_t:              104,      0,     980,     208,    1192,   0,   0
zil_lwb_cache:          192,      0,       1,      39,       4,   0,   0
zfs_znode_cache:        400,      0,     334,     134,     408,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           6          0
irq20: atapci0                   4513303      98115
irq23: ehci0 ehci1                  7747        168
cpu0:timer                       2138595      46491
irq256: re0                        10255        222
cpu1:timer                         62609       1361
cpu4:timer                       1596290      34701
cpu2:timer                       1677552      36468
cpu5:timer                        544898      11845
cpu7:timer                       1247772      27125
cpu3:timer                        252423       5487
cpu6:timer                        937091      20371
Total                           12988541     282359

------------------------------------------------------------------------
pstat -T

111/12328 files
0M/32767M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/label/swap0   33554168        0 33554168     0%
/dev/label/swap1   33554168        0 33554168     0%
Total             67108336        0 67108336     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             ad4              ad6             cpu
  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 126.57 49574 6127.41  127.69 49139 6127.67   0  0  1  0 99

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

nfsstat: new client/server not loaded

------------------------------------------------------------------------
netstat -s

tcp:
	1925 packets sent
		1863 data packets (283378 bytes)
		104 data packets (129984 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		51 ack-only packets (30 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		10 control packets
	2967 packets received
		1815 acks (for 282176 bytes)
		1 duplicate ack
		0 acks for unsent data
		1196 packets (70036 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	7 connection requests
	4 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	1 ignored RSTs in the window
	6 connections established (including accepts)
	34 connections closed (including 2 drops)
		2 connections updated cached RTT on close
		4 connections updated cached RTT variance on close
		2 connections updated cached ssthresh on close
	1 embryonic connection dropped
	1815 segments updated rtt (of 1202 attempts)
	24 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	921 correct ACK header predictions
	651 correct data packet header predictions
	5 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		4 completed
		0 bucket overflow
		0 cache overflow
		1 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	5 cookies sent
	0 cookies received
	3 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	1 SACK option (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	192 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	192 delivered
	206 datagrams output
	0 times multicast source filter matched
ip:
	3782 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	3763 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	19 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	2836 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	104 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	0 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		echo reply: 657
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		echo: 657
	657 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
pfsync:
	0 packets received (IPv4)
	0 packets received (IPv6)
		0 packets discarded for bad interface
		0 packets discarded for bad ttl
		0 packets shorter than header
		0 packets discarded for bad version
		0 packets discarded for bad HMAC
		0 packets discarded for bad action
		0 packets discarded for short packet
		0 states discarded for bad values
		0 stale states
		0 failed state lookup/inserts
	0 packets sent (IPv4)
	0 packets sent (IPv6)
		20 send failed due to mbuf memory error
		0 send error
arp:
	6 ARP requests sent
	1 ARP reply sent
	2 ARP requests received
	3 ARP replies received
	5 ARP packets received
	0 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	127 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	81 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	161 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Input histogram:
		TCP: 16
		UDP: 37
		ICMP6: 74
	Mbuf statistics:
		14 one mbuf
		113 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
		87 first candidate
		3 same address
		85 appropriate scope
		3 outgoing interface
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		echo: 30
		echo reply: 7
		neighbor solicitation: 44
		neighbor advertisement: 2
		MLDv2 listener report: 9
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Input histogram:
		echo: 7
		echo reply: 28
		neighbor solicitation: 2
		neighbor advertisement: 37
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	7 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output

------------------------------------------------------------------------
netstat -m

278/2689/2967 mbufs in use (current/cache/total)
246/1932/2178/25600 mbuf clusters in use (current/cache/total/max)
257/1674 mbuf+clusters out of packet secondary zone in use (current/cache)
0/54/54/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
561K/4752K/5313K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
usbus     0 <Link#1>                               0     0     0        0     0     0    0 
re0    1500 <Link#2>      54:04:a6:7e:ef:62     3916     0     0     2903     0     0    0 
re0    1500 fe80::5604:a6 fe80::5604:a6ff:f        1     -     -        3     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2       32     -     -       37     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2       21     -     -       14     -     -    - 
re0    1500 176.9.47.224/ static.245.47.9.1     1809     -     -     1237     -     -    - 
usbus     0 <Link#3>                               0     0     0        0     0     0    0 
pflog 33152 <Link#4>                               0     0     0        0     0     0    0 
pfsyn  1500 <Link#5>                               0     0     0        0    20     0    0 
lo0   16384 <Link#6>                               0     0     0        0     0     0    0 
lo0   16384 localhost     ::1                      0     -     -       14     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0   16384 your-net      localhost                0     -     -        0     -     -    - 
lo0   16384 127.0.1.1/32  mail                    14     -     -        0     -     -    - 
lo0   16384 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            176.9.47.225       UGS         2     1237    re0
127.0.0.1          link#6             UH          0        0    lo0
127.0.1.1          link#6             UH          0        0    lo0
176.9.47.224/27    link#2             U           0        0    re0
176.9.47.245       link#2             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0 =>
default                           2a01:4f8:150:63e0::1          UGS         re0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2a01:4f8:150:63e0::/59            link#2                        U           re0
2a01:4f8:150:63e2::2              link#2                        UHS         lo0
2a01:4f8:150:63e2::3              link#2                        UHS         lo0
2a01:4f8:150:63e2::4              link#6                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#2                        U           re0
fe80::5604:a6ff:fe7e:ef62%re0     link#2                        UHS         lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
ff01::%re0/32                     fe80::5604:a6ff:fe7e:ef62%re0 U           re0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re0/32                     fe80::5604:a6ff:fe7e:ef62%re0 U           re0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffffe0011774b70 tcp4       0      0 127.0.1.1.25       *.*                LISTEN
fffffe00116aeb70 tcp4      96      0 176.9.47.245.22    88.69.228.83.57255 ESTABLISHED
fffffe0011757000 tcp4       0      0 127.0.0.1.25       *.*                LISTEN
fffffe00117743d0 tcp4       0      0 127.0.0.1.22       *.*                LISTEN
fffffe00117747a0 tcp4       0      0 176.9.47.245.22    *.*                LISTEN
fffffe00114a47a8 udp6       0      0 *.*                *.*                
fffffe0011490ab8 udp6       0      0 2a01:4f8:150:63e.1 *.*                
fffffe00112fe000 udp4       0      0 127.0.1.1.514      *.*                
fffffe00112feab8 udp6       0      0 2a01:4f8:150:63e.5 *.*                
fffffe00114a4000 udp4       0      0 127.0.1.1.123      *.*                
fffffe00113977a8 udp4       0      0 176.9.47.245.123   *.*                
fffffe0011313dc8 udp6       0      0 fe80:2::5604:a6f.1 *.*                
fffffe00112fe620 udp6       0      0 2a01:4f8:150:63e.1 *.*                
fffffe0011397310 udp6       0      0 2a01:4f8:150:63e.1 *.*                
fffffe0011490188 udp4       0      0 127.0.0.1.123      *.*                
fffffe0011490310 udp6       0      0 fe80:6::1.123      *.*                
fffffe0011490620 udp6       0      0 ::1.123            *.*                
fffffe0011491188 udp6       0      0 *.123              *.*                
fffffe0011491310 udp4       0      0 *.123              *.*                
fffffe0011313930 udp4       0      0 *.514              *.*                
fffffe00113137a8 udp6       0      0 *.514              *.*                
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe001143f2d0 stream      0      0 fffffe0011b9f000        0        0        0 /tmp/tmux-0/default
fffffe001143f3c0 stream      0      0        0 fffffe001143f4b0        0        0
fffffe001143f4b0 stream      0      0        0 fffffe001143f3c0        0        0
fffffe001143f000 stream      0      0        0 fffffe001143ee10        0        0
fffffe001143ee10 stream      0      0        0 fffffe001143f000        0        0
fffffe001143ed20 stream      0      0        0        0        0        0
fffffe001143ec30 stream      0      0        0 fffffe001143eb40        0        0
fffffe001143eb40 stream      0      0        0 fffffe001143ec30        0        0
fffffe00114971e0 stream      0      0 fffffe0011eb2780        0        0        0 /tmp/ssh-cLcoCcoRFd/agent.3947
fffffe001143e960 stream      0      0 fffffe001135e000        0        0        0 /var/run/devd.pipe
fffffe0011f820f0 dgram       0      0        0 fffffe0011e39d20        0 fffffe0011e39e10
fffffe0011f82000 dgram       0      0        0 fffffe0011e39c30        0        0
fffffe0011e39e10 dgram       0      0        0 fffffe0011e39d20        0        0
fffffe0011e39d20 dgram       0      0 fffffe00117fd1e0        0 fffffe0011f820f0        0 /var/run/logpriv
fffffe0011e39c30 dgram       0      0 fffffe00117f9d20        0 fffffe0011f82000        0 /var/run/log
fffffe001143fe10 dgram       0      0        0 fffffe00114a7960        0 fffffe00114a75a0
fffffe00114a74b0 dgram       0      0        0 fffffe00114a7a50        0        0
fffffe00114a75a0 dgram       0      0        0 fffffe00114a7960        0 fffffe00114a7870
fffffe00114a7870 dgram       0      0        0 fffffe00114a7960        0        0
fffffe00114a7960 dgram       0      0 fffffe0004e9b1e0        0 fffffe001143fe10        0 /var/run/logpriv
fffffe00114a7a50 dgram       0      0 fffffe0004e9b3c0        0 fffffe00114a74b0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address         
tcp4  0/0/10         mail.smtp              
tcp4  0/0/10         localhost.smtp         
tcp4  0/0/128        localhost.ssh          
tcp4  0/0/128        static.245.47.9..ssh   
unix  0/0/16         /tmp/tmux-0/default
unix  0/0/128        /tmp/ssh-cLcoCcoRFd/agent.3947
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     csh         5768 root /             2 drwxr-xr-x     512  r
root     csh         5768   wd /             6 drwxr-xr-x     512  r
root     csh         5768 text /         16524 -r-xr-xr-x  369072  r
root     csh         5768 ctty /dev        114 crw--w----   pts/3 rw
root     csh         5768   15 /dev        114 crw--w----   pts/3 rw
root     csh         5768   16 /dev        114 crw--w----   pts/3 rw
root     csh         5768   17 /dev        114 crw--w----   pts/3 rw
root     csh         5768   18 /dev        114 crw--w----   pts/3 rw
root     csh         5768   19 /dev        114 crw--w----   pts/3 rw
root     csh         5736 root /usr/jails/mail      3 ?---------  18446744071577938073  r
root     csh         5736   wd /usr/jails/mail      3 ?---------  18446744071577938073  r
root     csh         5736 jail /usr/jails/mail      3 ?---------  18446744071577938073  r
root     csh         5736 text /usr/jails/mail   2993 ?---------       0  r
root     csh         5736 ctty /dev        111 crw--w----   pts/0 rw
root     csh         5736   15 /dev        111 crw--w----   pts/0 rw
root     csh         5736   16 /dev        111 crw--w----   pts/0 rw
root     csh         5736   17 /dev        111 crw--w----   pts/0 rw
root     csh         5736   18 /dev        111 crw--w----   pts/0 rw
root     csh         5736   19 /dev        111 crw--w----   pts/0 rw
root     cron        5709 root /usr/jails/mail      3 ?---------  18446744071577938073  r
root     cron        5709   wd /usr/jails/mail     55 ?---------  18446744071577938073  r
root     cron        5709 jail /usr/jails/mail      3 ?---------  18446744071577938073  r
root     cron        5709 text /usr/jails/mail  11173 ?---------  18446744071577938073  r
root     cron        5709    0 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     cron        5709    1 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     cron        5709    2 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     cron        5709    3 /usr/jails/mail 159843 ?---------  18446744071577938073  w
root     cron        5709    4* local dgram fffffe0011f820f0 <-> fffffe0011e39d20
smmsp    sendmail    5702 root /usr/jails/mail      3 ?---------  18446744071577938073  r
smmsp    sendmail    5702   wd /usr/jails/mail    880 ?---------  18446744071577938073  r
smmsp    sendmail    5702 jail /usr/jails/mail      3 ?---------  18446744071577938073  r
smmsp    sendmail    5702 text /usr/jails/mail  11727 ?---------  18446744071577938073  r
smmsp    sendmail    5702    0 /usr/jails/mail/dev      6 crw-rw-rw-    null  r
smmsp    sendmail    5702    1 /usr/jails/mail/dev      6 crw-rw-rw-    null  w
smmsp    sendmail    5702    2 /usr/jails/mail/dev      6 crw-rw-rw-    null  w
smmsp    sendmail    5702    3* local dgram fffffe0011f82000 <-> fffffe0011e39c30
smmsp    sendmail    5702    4 /usr/jails/mail 159842 ?---------  18446744071577938073  w
root     sendmail    5698 root /usr/jails/mail      3 ?---------  18446744071577938073  r
root     sendmail    5698   wd /usr/jails/mail     80 ?---------  18446744071577938073  r
root     sendmail    5698 jail /usr/jails/mail      3 ?---------  18446744071577938073  r
root     sendmail    5698 text /usr/jails/mail  11727 ?---------  18446744071577938073  r
root     sendmail    5698    0 /usr/jails/mail/dev      6 crw-rw-rw-    null  r
root     sendmail    5698    1 /usr/jails/mail/dev      6 crw-rw-rw-    null  w
root     sendmail    5698    2 /usr/jails/mail/dev      6 crw-rw-rw-    null  w
root     sendmail    5698    3* internet stream tcp fffffe0011774b70
root     sendmail    5698    4* local dgram fffffe0011e39e10 <-> fffffe0011e39d20
root     sendmail    5698    5 /usr/jails/mail 159841 ?---------  18446744071577938073  w
root     syslogd     5571 root /usr/jails/mail      3 ?---------  18446744071577938073  r
root     syslogd     5571   wd /usr/jails/mail      3 ?---------  18446744071577938073  r
root     syslogd     5571 jail /usr/jails/mail      3 ?---------  18446744071577938073  r
root     syslogd     5571 text /usr/jails/mail  11765 ?---------  18446744071577938073  r
root     syslogd     5571    0 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     syslogd     5571    1 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     syslogd     5571    2 /usr/jails/mail/dev      6 crw-rw-rw-    null rw
root     syslogd     5571    3 /usr/jails/mail 159829 ?---------  18446744071577938073  w
root     syslogd     5571    4* local dgram fffffe0011e39c30
root     syslogd     5571    5* local dgram fffffe0011e39d20
root     syslogd     5571    6* internet6 dgram udp fffffe00112feab8
root     syslogd     5571    7* internet dgram udp fffffe00112fe000
root     syslogd     5571    9 /usr/jails/mail  12973 ?---------  18446744071577938073  w
root     syslogd     5571   10 /usr/jails/mail  12975 ?---------  18446744071577938073  w
root     syslogd     5571   11 /usr/jails/mail  12968 ?---------  18446744071577938073  w
root     syslogd     5571   12 /usr/jails/mail 160753 ?---------  18446744071577938073  w
root     syslogd     5571   13 /usr/jails/mail  12971 ?---------  18446744071577938073  w
root     syslogd     5571   14 /usr/jails/mail  12976 ?---------  18446744071577938073  w
root     syslogd     5571   15 /usr/jails/mail 160636 ?---------       0  w
root     syslogd     5571   16 /usr/jails/mail  12970 ?---------  18446744071577938073  w
root     syslogd     5571   17 /usr/jails/mail  12974 ?---------  18446744071577938073  w
root     less        4569 root /             2 drwxr-xr-x     512  r
root     less        4569   wd /             6 drwxr-xr-x     512  r
root     less        4569 text /usr     106774 -r-xr-xr-x  144656  r
root     less        4569 ctty /dev        110 crw--w----   pts/2 rw
root     less        4569    0* pipe fffffe0011efa888 <-> fffffe0011efa9e0      0 rw
root     less        4569    1 /dev        110 crw--w----   pts/2 rw
root     less        4569    2 /dev        110 crw--w----   pts/2 rw
root     less        4569    3 /dev        110 crw--w----   pts/2  r
root     sh          4559 root /             2 drwxr-xr-x     512  r
root     sh          4559   wd /             6 drwxr-xr-x     512  r
root     sh          4559 text /         16578 -r-xr-xr-x  142960  r
root     sh          4559 ctty /dev        110 crw--w----   pts/2 rw
root     sh          4559    0 /dev        110 crw--w----   pts/2 rw
root     sh          4559    1 /dev        110 crw--w----   pts/2 rw
root     sh          4559    2 /dev        110 crw--w----   pts/2 rw
root     sh          4559   10 /usr     106821 -r-xr-xr-x   21019  r
root     csh         4555 root /             2 drwxr-xr-x     512  r
root     csh         4555   wd /             6 drwxr-xr-x     512  r
root     csh         4555 text /         16524 -r-xr-xr-x  369072  r
root     csh         4555 ctty /dev        110 crw--w----   pts/2 rw
root     csh         4555   15 /dev        110 crw--w----   pts/2 rw
root     csh         4555   16 /dev        110 crw--w----   pts/2 rw
root     csh         4555   17 /dev        110 crw--w----   pts/2 rw
root     csh         4555   18 /dev        110 crw--w----   pts/2 rw
root     csh         4555   19 /dev        110 crw--w----   pts/2 rw
root     csh         4025 root /             2 drwxr-xr-x     512  r
root     csh         4025   wd /             6 drwxr-xr-x     512  r
root     csh         4025 text /         16524 -r-xr-xr-x  369072  r
root     csh         4025 ctty /dev        111 crw--w----   pts/0 rw
root     csh         4025   15 /dev        111 crw--w----   pts/0 rw
root     csh         4025   16 /dev        111 crw--w----   pts/0 rw
root     csh         4025   17 /dev        111 crw--w----   pts/0 rw
root     csh         4025   18 /dev        111 crw--w----   pts/0 rw
root     csh         4025   19 /dev        111 crw--w----   pts/0 rw
root     tmux        4024 root /             2 drwxr-xr-x     512  r
root     tmux        4024   wd /             6 drwxr-xr-x     512  r
root     tmux        4024 text /usr     103415 -r-xr-xr-x  327240  r
root     tmux        4024    0 /dev          6 crw-rw-rw-    null rw
root     tmux        4024    1 /dev          6 crw-rw-rw-    null rw
root     tmux        4024    2 /dev          6 crw-rw-rw-    null rw
root     tmux        4024    3* local stream fffffe001143f4b0 <-> fffffe001143f3c0
root     tmux        4024    4* local stream fffffe001143f3c0 <-> fffffe001143f4b0
root     tmux        4024    5* local stream fffffe001143ed20
root     tmux        4024    6* local stream fffffe001143f2d0
root     tmux        4024    7* local stream fffffe001143f000 <-> fffffe001143ee10
root     tmux        4024    8 /dev        113 crw--w----   pts/1 rw
root     tmux        4024    9 /dev        113 crw--w----   pts/1 rw
root     tmux        4024   10 /dev        113 crw--w----   pts/1 rw
root     tmux        4024   11 /dev        113 crw--w----   pts/1 rw
root     tmux        4024   12* pseudo-terminal master      pts/0 rw
root     tmux        4024   13* pseudo-terminal master      pts/2 rw
root     tmux        4024   14* pseudo-terminal master      pts/3 rw
root     tmux        4022 root /             2 drwxr-xr-x     512  r
root     tmux        4022   wd /             6 drwxr-xr-x     512  r
root     tmux        4022 text /usr     103415 -r-xr-xr-x  327240  r
root     tmux        4022 ctty /dev        113 crw--w----   pts/1 rw
root     tmux        4022    0 /dev        113 crw--w----   pts/1 rw
root     tmux        4022    1 /dev        113 crw--w----   pts/1 rw
root     tmux        4022    2 /dev        113 crw--w----   pts/1 rw
root     tmux        4022    3* local stream fffffe001143eb40 <-> fffffe001143ec30
root     tmux        4022    4* local stream fffffe001143ec30 <-> fffffe001143eb40
root     tmux        4022    5* local stream fffffe001143ed20
root     tmux        4022    6* local stream fffffe001143ee10 <-> fffffe001143f000
root     csh         3949 root /             2 drwxr-xr-x     512  r
root     csh         3949   wd /             6 drwxr-xr-x     512  r
root     csh         3949 text /         16524 -r-xr-xr-x  369072  r
root     csh         3949 ctty /dev        113 crw--w----   pts/1 rw
root     csh         3949   15 /dev        113 crw--w----   pts/1 rw
root     csh         3949   16 /dev        113 crw--w----   pts/1 rw
root     csh         3949   17 /dev        113 crw--w----   pts/1 rw
root     csh         3949   18 /dev        113 crw--w----   pts/1 rw
root     csh         3949   19 /dev        113 crw--w----   pts/1 rw
root     sshd        3947 root /             2 drwxr-xr-x     512  r
root     sshd        3947   wd /             2 drwxr-xr-x     512  r
root     sshd        3947 text /usr     147507 -r-xr-xr-x  269712  r
root     sshd        3947    0 /dev          6 crw-rw-rw-    null rw
root     sshd        3947    1 /dev          6 crw-rw-rw-    null rw
root     sshd        3947    2 /dev          6 crw-rw-rw-    null rw
root     sshd        3947    3* internet stream tcp fffffe00116aeb70
root     sshd        3947    4* pipe fffffe00113032d8 <-> fffffe0011303430      0 rw
root     sshd        3947    5* pipe fffffe0011303430 <-> fffffe00113032d8      0 rw
root     sshd        3947    6* local stream fffffe00114971e0
root     sshd        3947    7* pseudo-terminal master      pts/1 rw
root     sshd        3947    9* pseudo-terminal master      pts/1 rw
root     sshd        3947   10* pseudo-terminal master      pts/1 rw
_dhcp    dhclient    3946 root /var      16386 dr-xr-xr-x     512  r
_dhcp    dhclient    3946   wd /var      16386 dr-xr-xr-x     512  r
_dhcp    dhclient    3946 jail /var      16386 dr-xr-xr-x     512  r
_dhcp    dhclient    3946 text /         16581 -r-xr-xr-x   92376  r
_dhcp    dhclient    3946    0 /dev          6 crw-rw-rw-    null rw
_dhcp    dhclient    3946    1 /dev          6 crw-rw-rw-    null rw
_dhcp    dhclient    3946    2 /dev          6 crw-rw-rw-    null rw
_dhcp    dhclient    3946    3* local dgram fffffe001143fe10 <-> fffffe00114a7960
_dhcp    dhclient    3946    5* route raw 0 fffffe00114a22a8
_dhcp    dhclient    3946    6* pipe fffffe0011ef9430 <-> fffffe0011ef92d8      0 rw
_dhcp    dhclient    3946    7 /var      24594 ----------    1072  w
_dhcp    dhclient    3946    8 /dev         23 crw-------     bpf rw
_dhcp    dhclient    3946    9* internet raw ip fffffe0011effdc8
root     dhclient    3894 root /             2 drwxr-xr-x     512  r
root     dhclient    3894   wd /             2 drwxr-xr-x     512  r
root     dhclient    3894 text /         16581 -r-xr-xr-x   92376  r
root     dhclient    3894    0 /dev          6 crw-rw-rw-    null rw
root     dhclient    3894    1 /dev          6 crw-rw-rw-    null rw
root     dhclient    3894    2 /dev          6 crw-rw-rw-    null rw
root     dhclient    3894    3* local dgram fffffe001143fe10 <-> fffffe00114a7960
root     dhclient    3894    5* pipe fffffe0011ef92d8 <-> fffffe0011ef9430      0 rw
root     getty       1915 root /             2 drwxr-xr-x     512  r
root     getty       1915   wd /             2 drwxr-xr-x     512  r
root     getty       1915 text /usr       4044 -r-xr-xr-x   27528  r
root     getty       1915 ctty /dev         35 crw-------   ttyv0 rw
root     getty       1915    0 /dev         35 crw-------   ttyv0 rw
root     getty       1915    1 /dev         35 crw-------   ttyv0 rw
root     getty       1915    2 /dev         35 crw-------   ttyv0 rw
root     cron        1460 root /             2 drwxr-xr-x     512  r
root     cron        1460   wd /var      16385 drwxr-x---     512  r
root     cron        1460 text /usr     147573 -r-xr-xr-x   41384  r
root     cron        1460    0 /dev          6 crw-rw-rw-    null rw
root     cron        1460    1 /dev          6 crw-rw-rw-    null rw
root     cron        1460    2 /dev          6 crw-rw-rw-    null rw
root     cron        1460    3 /var      16466 -rw-------       4  w
smmsp    sendmail    1454 root /             2 drwxr-xr-x     512  r
smmsp    sendmail    1454   wd /var      16402 drwxrwx---     512  r
smmsp    sendmail    1454 text /usr      20443 -r-xr-sr-x  707520  r
smmsp    sendmail    1454    0 /dev          6 crw-rw-rw-    null  r
smmsp    sendmail    1454    1 /dev          6 crw-rw-rw-    null  w
smmsp    sendmail    1454    2 /dev          6 crw-rw-rw-    null  w
smmsp    sendmail    1454    3* local dgram fffffe00114a74b0 <-> fffffe00114a7a50
smmsp    sendmail    1454    4 /var      16460 -rw-------      50  w
root     sendmail    1450 root /             2 drwxr-xr-x     512  r
root     sendmail    1450   wd /var      16398 drwxr-xr-x     512  r
root     sendmail    1450 text /usr      20443 -r-xr-sr-x  707520  r
root     sendmail    1450    0 /dev          6 crw-rw-rw-    null  r
root     sendmail    1450    1 /dev          6 crw-rw-rw-    null  w
root     sendmail    1450    2 /dev          6 crw-rw-rw-    null  w
root     sendmail    1450    3* internet stream tcp fffffe0011757000
root     sendmail    1450    4* local dgram fffffe00114a75a0 <-> fffffe00114a7960
root     sendmail    1450    5 /var      16459 -rw-------      79  w
root     sshd        1422 root /             2 drwxr-xr-x     512  r
root     sshd        1422   wd /             2 drwxr-xr-x     512  r
root     sshd        1422 text /usr     147507 -r-xr-xr-x  269712  r
root     sshd        1422    0 /dev          6 crw-rw-rw-    null rw
root     sshd        1422    1 /dev          6 crw-rw-rw-    null rw
root     sshd        1422    2 /dev          6 crw-rw-rw-    null rw
root     sshd        1422    3* internet stream tcp fffffe00117747a0
root     sshd        1422    4* internet stream tcp fffffe00117743d0
root     ntpd        1369 root /             2 drwxr-xr-x     512  r
root     ntpd        1369   wd /             2 drwxr-xr-x     512  r
root     ntpd        1369 text /usr     142430 -r-xr-xr-x  391784  r
root     ntpd        1369    0 /dev          6 crw-rw-rw-    null rw
root     ntpd        1369    1 /dev          6 crw-rw-rw-    null rw
root     ntpd        1369    2 /dev          6 crw-rw-rw-    null rw
root     ntpd        1369    3* local dgram fffffe00114a7870 <-> fffffe00114a7960
root     ntpd        1369    4* internet6 dgram udp fffffe00114a47a8
root     ntpd        1369   20* internet dgram udp fffffe0011491310
root     ntpd        1369   21* internet6 dgram udp fffffe0011491188
root     ntpd        1369   22* internet6 dgram udp fffffe0011397310
root     ntpd        1369   23* internet6 dgram udp fffffe00112fe620
root     ntpd        1369   24* internet6 dgram udp fffffe0011313dc8
root     ntpd        1369   25* internet6 dgram udp fffffe0011490620
root     ntpd        1369   26* internet6 dgram udp fffffe0011490310
root     ntpd        1369   27* internet dgram udp fffffe0011490188
root     ntpd        1369   28* route raw 0 fffffe001148e2a8
root     ntpd        1369   29* internet dgram udp fffffe00113977a8
root     ntpd        1369   30* internet dgram udp fffffe00114a4000
root     ntpd        1369   31* internet6 dgram udp fffffe0011490ab8
root     syslogd     1195 root /             2 drwxr-xr-x     512  r
root     syslogd     1195   wd /             2 drwxr-xr-x     512  r
root     syslogd     1195 text /usr     153163 -r-xr-xr-x   39832  r
root     syslogd     1195    0 /dev          6 crw-rw-rw-    null rw
root     syslogd     1195    1 /dev          6 crw-rw-rw-    null rw
root     syslogd     1195    2 /dev          6 crw-rw-rw-    null rw
root     syslogd     1195    3 /var      16450 -rw-------       4  w
root     syslogd     1195    4* local dgram fffffe00114a7a50
root     syslogd     1195    5* local dgram fffffe00114a7960
root     syslogd     1195    6* internet6 dgram udp fffffe00113137a8
root     syslogd     1195    7* internet dgram udp fffffe0011313930
root     syslogd     1195    8 /dev         15 crw-------    klog  r
root     syslogd     1195   10 /dev          5 crw-------  console  w
root     syslogd     1195   11 /var       8968 -rw-r--r--    2169  w
root     syslogd     1195   12 /var       8228 -rw-------      62  w
root     syslogd     1195   13 /var       8225 -rw-------    4540  w
root     syslogd     1195   14 /var       8237 -rw-r-----   49972  w
root     syslogd     1195   15 /var       8224 -rw-r--r--      62  w
root     syslogd     1195   16 /var       8229 -rw-------    3345  w
root     syslogd     1195   17 /var       8221 -rw-------   93187  w
root     syslogd     1195   18 /var       8223 -rw-------      62  w
root     syslogd     1195   19 /var       8227 -rw-r-----      62  w
root     devd         957 root /             2 drwxr-xr-x     512  r
root     devd         957   wd /             2 drwxr-xr-x     512  r
root     devd         957 text /         16573 -r-xr-xr-x  424288  r
root     devd         957    0 /dev          6 crw-rw-rw-    null rw
root     devd         957    1 /dev          6 crw-rw-rw-    null rw
root     devd         957    2 /dev          6 crw-rw-rw-    null rw
root     devd         957    3 /dev          4 crw-------  devctl  r
root     devd         957    4* local stream fffffe001143e960
root     devd         957    5 /var      16437 -rw-------       3  w
root     zfskern       42 root /             2 drwxr-xr-x     512  r
root     zfskern       42   wd /             2 drwxr-xr-x     512  r
root     init           1 root /             2 drwxr-xr-x     512  r
root     init           1   wd /             2 drwxr-xr-x     512  r
root     init           1 text /         16606 -r-xr-xr-x  773568  r
root     kernel         0 root /             2 drwxr-xr-x     512  r
root     kernel         0   wd /             2 drwxr-xr-x     512  r

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RC1 #10: Sun Oct 30 11:34:41 UTC 2011
    root@phallus.wzff.de:/usr/obj/usr/src/sys/PHALLUS amd64
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (3411.56-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x17bae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16434438144 (15673 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <ALASKA A M I> on motherboard
ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20110527/psargs-392)
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20110527/nsinit-380)
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
pci0: <simple comms> at device 22.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe503000-0xfe5033ff irq 23 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0: <EHCI (generic) USB 2.0 controller> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <serial bus, USB> at device 0.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 18 at device 0.0 on pci4
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 54:04:a6:7e:ef:62
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe502000-0xfe5023ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1: <EHCI (generic) USB 2.0 controller> on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel Cougar Point SATA300 controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xfe501000-0xfe5017ff irq 20 at device 31.2 on pci0
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.30 controller with 6 6Gbps ports, PM not supported
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROM> at iomem 0xcd800-0xce7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad4: 2861588MB <Seagate ST33000651AS CC45> at ata2-master UDMA100 SATA 6Gb/s
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ad6: 2861588MB <Seagate ST33000651AS CC45> at ata3-master UDMA100 SATA 6Gb/s
SMP: AP CPU #1 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
GEOM_MIRROR: Device mirror/root launched (2/2).
GEOM_MIRROR: Device mirror/usr launched (2/2).
GEOM_MIRROR: Device mirror/var launched (2/2).
GEOM_MIRROR: Device mirror/tmp launched (2/2).
GEOM_MIRROR: Device mirror/zfs launched (1/2).
GEOM_MIRROR: Device zfs: rebuilding provider ad4p7.
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
Trying to mount root from ufs:/dev/mirror/root [rw]...
Setting hostuuid: 2062ea40-5bcb-11d9-98a3-5404a67eef62.
Setting hostid: 0x832a19a6.
ZFS filesystem version 5
ZFS storage pool version 28
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/mirror/root: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/root: clean, 26782 free (2318 frags, 3058 blocks, 1.8% fragmentation)
/dev/mirror/var: DEFER FOR BACKGROUND CHECKING
/dev/mirror/tmp: DEFER FOR BACKGROUND CHECKING
/dev/mirror/usr: DEFER FOR BACKGROUND CHECKING
Mounting local file systems:
WARNING: /var was not properly dismounted
WARNING: /var: GJOURNAL flag on fs but no gjournal provider below
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
.
Setting hostname: phallus.wzff.de.
/etc/rc: WARNING: $ipv6_ifconfig_re0 is obsolete.   Use ifconfig_re0_ipv6 instead.
Starting Network: lo0 re0 pflog0 pfsync0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
	ether 54:04:a6:7e:ef:62
	inet6 fe80::5604:a6ff:fe7e:ef62%re0 prefixlen 64 scopeid 0x2 
	inet6 2a01:4f8:150:63e2::2 prefixlen 59 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (none)
	status: no carrier
pflog0: flags=0<> metric 0 mtu 33152
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	syncpeer: 0.0.0.0 maxupd: 128
Starting devd.
Starting Network: usbus0.
Starting Network: usbus1.
Starting Network: pflog0.
pflog0: flags=0<> metric 0 mtu 33152
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Starting Network: pfsync0.
pfsync0: flags=0<> metric 0 mtu 1500
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	syncpeer: 0.0.0.0 maxupd: 128
Enabling pf.
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net default: gateway 2a01:4f8:150:63e0::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Waiting 30s for the default route interface: .(re0)
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
No core dumps found.
Clearing /tmp (X related).
Updating motd:.
Starting ntpd.
Starting sshd.
Configuring syscons: blanktime.
Starting cron.
Configuring jails:.
Starting jails: mail.wzff.de.
Starting background file system checks in 60 seconds.

Sun Oct 30 21:57:34 UTC 2011
Oct 30 22:19:58 phallus dhclient[1105]: connection closed
Oct 30 22:19:58 phallus dhclient[1105]: exiting.
Oct 30 22:19:59 phallus ntpd[1369]: sendto(141.40.103.103) (fd=24): No route to host
re0: link state changed to DOWN
Oct 30 22:20:01 phallus ntpd[1369]: bind() fd 22, family AF_INET6, port 123, scope 2, addr fe80::5604:a6ff:fe7e:ef62, mcast=0 flags=0x11 fails: Can't assign requested address
Oct 30 22:20:01 phallus ntpd[1369]: unable to create socket on re0 (9) for fe80::5604:a6ff:fe7e:ef62#123
re0: link state changed to UP
Oct 30 22:20:58 phallus dhclient[3018]: My address (176.9.47.245) was deleted, dhclient exiting
Oct 30 22:20:58 phallus dhclient[2966]: connection closed
Oct 30 22:20:58 phallus dhclient[2966]: exiting.
re0: link state changed to DOWN
Oct 30 22:21:01 phallus ntpd[1369]: bind() fd 22, family AF_INET6, port 123, scope 2, addr fe80::5604:a6ff:fe7e:ef62, mcast=0 flags=0x11 fails: Can't assign requested address
Oct 30 22:21:01 phallus ntpd[1369]: unable to create socket on re0 (13) for fe80::5604:a6ff:fe7e:ef62#123
re0: link state changed to UP
ifa_add_loopback_route: insertion failed


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address	= 0x360
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff804c4b9a
stack pointer	        = 0x28:0xffffff845da34780
frame pointer	        = 0x28:0xffffff845da347b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1369 (ntpd)
trap number		= 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff804fb9de at kdb_backtrace+0x5e
#1 0xffffffff804c6537 at panic+0x187
#2 0xffffffff8073a060 at trap_fatal+0x290
#3 0xffffffff8073a3a9 at trap_pfault+0x1f9
#4 0xffffffff8073a86f at trap+0x3df
#5 0xffffffff80724d9f at calltrap+0x8
#6 0xffffffff806530aa at in6_setscope+0x23a
#7 0xffffffff8063bdbb at in6_selectsrc+0x36b
#8 0xffffffff80639309 at in6_pcbladdr+0x149
#9 0xffffffff806399c4 at in6_pcbconnect_mbuf+0x44
#10 0xffffffff80656844 at udp6_connect+0x1a4
#11 0xffffffff80536e51 at kern_connect+0x121
#12 0xffffffff80536fc1 at sys_connect+0x41
#13 0xffffffff80739950 at amd64_syscall+0x450
#14 0xffffffff80725087 at Xfast_syscall+0xf7
GEOM_MIRROR: Device zfs: rebuilding provider ad4p7 stopped.
Uptime: 36m34s
Dumping 753 out of 16284 MB:..3%..11%..22%..32%..41%..51%..62%..71%..81%..92%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	PHALLUS
machine	amd64
cpu	HAMMER
makeoptions	DEBUG=-g
options	RCTL
options	RACCT
options	ALTQ_NOPCC
options	ALTQ_PRIQ
options	ALTQ_HFSC
options	ALTQ_RIO
options	ALTQ_RED
options	ALTQ_CBQ
options	ALTQ
options	USB_DEBUG
options	ATA_STATIC_ID
options	SMP
options	KDB_TRACE
options	KDB
options	INCLUDE_CONFIG_FILE
options	FLOWTABLE
options	MAC
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	P1003_1B_SEMAPHORES
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	COMPAT_43TTY
options	GEOM_LABEL
options	GEOM_MIRROR
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	NFSLOCKD
options	NFSSERVER
options	NFSCLIENT
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	ata
device	atadisk
device	ataraid
device	atapicd
device	atapifd
device	atapist
device	atkbdc
device	atkbd
device	kbdmux
device	vga
device	sc
device	uart
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	dc
device	et
device	fxp
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	loop
device	random
device	ether
device	vlan
device	tun
device	pty
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	usb
device	uhid
device	ukbd
device	ulpt
device	umass
device	ums
device	urio
device	scbus
device	pf
device	pflog
device	pfsync

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist


kgdb backtrace from the second crash:

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff804bd037
stack pointer	        = 0x28:0xffffff845dbec9d0
frame pointer	        = 0x28:0xffffff845dbeca00
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 20982 (sbcl)
trap number		= 12
panic: page fault
cpuid = 6
KDB: stack backtrace:
#0 0xffffffff804fb08e at kdb_backtrace+0x5e
#1 0xffffffff804c5be7 at panic+0x187
#2 0xffffffff807392b0 at trap_fatal+0x290
#3 0xffffffff807395f9 at trap_pfault+0x1f9
#4 0xffffffff80739abf at trap+0x3df
#5 0xffffffff80723fef at calltrap+0x8
#6 0xffffffff804bd48f at racct_proc_exit+0x7f
#7 0xffffffff8049620f at fork1+0x35f
#8 0xffffffff80497c62 at sys_fork+0x22
#9 0xffffffff80738ba0 at amd64_syscall+0x450
#10 0xffffffff807242d7 at Xfast_syscall+0xf7
GEOM_MIRROR: Device zfs: rebuilding provider ad4p7 stopped.
Uptime: 2h38m43s
Dumping 1598 out of 16284 MB:..2%panic: bufwrite: buffer is not busy???
cpuid = 6
..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224		__asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff804c5725 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:442
#2  0xffffffff804c5bd1 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:607
#3  0xffffffff807392b0 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:818
#4  0xffffffff807395f9 in trap_pfault (frame=0xffffff845dbec920, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:734
#5  0xffffffff80739abf in trap (frame=0xffffff845dbec920) at /usr/src/sys/amd64/amd64/trap.c:473
#6  0xffffffff80723fef in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff804bd037 in racct_set_locked (p=0xfffffe005dfd1910, resource=0, amount=5843)
    at /usr/src/sys/kern/kern_racct.c:382
#8  0xffffffff804bd48f in racct_proc_exit (p=0xfffffe005dfd1910) at /usr/src/sys/kern/kern_racct.c:637
#9  0xffffffff8049620f in fork1 (td=0xfffffe005dfdb000, flags=20, pages=4, procp=0xffffff845dbecb10, procdescp=Variable "procdescp" is not available.
)
    at /usr/src/sys/kern/kern_fork.c:946
#10 0xffffffff80497c62 in sys_fork (td=0xfffffe005dfdb000, uap=Variable "uap" is not available.
) at /usr/src/sys/kern/kern_fork.c:110
#11 0xffffffff80738ba0 in amd64_syscall (td=0xfffffe005dfdb000, traced=0) at subr_syscall.c:131
#12 0xffffffff807242d7 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387
#13 0x0000000800d257ec in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

>Description:
	My system has been crashing in ip6 on 9.0-RC1 from a few days ago while
	compiling sbcl in a jail, and since I had lost the kernel debugging
	information and wanted to make sure it wasn't my fault, I upgraded to 9.0-RC2
	This time, it crashed in fork1, again with a page fault.
	
	At first, the jail had these limits, on which sbcl wouldn't compile, because
	it likes to mmap() a huge chunk of memory, which it doesn't completely use.

	jail:foo:memoryuse:deny=2g
	jail:foo:swapuse:deny=2g
	jail:foo:maxproc:deny=4096
	jail:foo:openfiles:deny=4096

	After using rctl -a jail:foo:swapuse:deny=10G, sbcl *seemed* to work, but
	then the server crashed both times. Since rctl is fairly new, I expect the
	bug to be there, but it could be something else entirely. I'm not exactly a
	kernel debugging expert, so I might need further instructions.

	The second time, it didn't generate a nice crash report, but I can at least
	provide a kgdb backtrace, and probably more if you tell me how.

>How-To-Repeat:
	Probably by compiling sbcl in a jail with the rctl limits mentioned above.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201111091631.pA9GVTpF061747>