Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 6 Mar 2013 12:50:01 GMT
From:      Rasmus Skaarup <freebsd@gal.dk>
To:        freebsd-bugs@FreeBSD.org
Subject:   Re: kern/176636: Periodical crashes with 9.1-R
Message-ID:  <201303061250.r26Co1jN079370@freefall.freebsd.org>

next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/176636; it has been noted by GNATS.

From: Rasmus Skaarup <freebsd@gal.dk>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/176636: Periodical crashes with 9.1-R
Date: Wed, 6 Mar 2013 13:40:53 +0100

 The system has become highly unstable now. It crashes very often, and =
 almost every time I issue a zpool command.=20
 
 I tried booting with the patched kernel, but it still crashes. Here is =
 the info from the latest crash:
 
 This command crashed the system:
 
 ***
 [root@dentredje rc.d]# zpool status -v
   pool: data2
  state: ONLINE
 status: One or more devices has experienced an error resulting in data
 	corruption.  Applications may be affected.
 action: Restore the file in question if possible.  Otherwise restore the
 	entire pool from backup.
    see: http://illumos.org/msg/ZFS-8000-8A
   scan: scrub repaired 0 in 0h15m with 8 errors on Wed Mar  6 11:28:12 =
 2013
 config:
 
 	NAME        STATE     READ WRITE CKSUM
 	data2       ONLINE       0     0     1
 	  vtbd1p1   ONLINE       0     0     2
 	  vtbd2p1   ONLINE       0     0     2
 
 errors: Permanent errors have been detected in the following files:
 
         data2/home:<0x0>
         =
 /data2/home/jacobk/www.secretsounds.dk/40days/final_tracks_for_5.1/bob_5/b=
 ob_5-start_05_01_46_11-5_of_5.wav
         /data2/home/skaarup/mysql-backup.sql
         data2/src:<0x0>
         data2/src:<0xb>
 ***
 
 info.txt.6:
 
 [root@dentredje crash]# cat info.6=20
 Dump header from device /dev/vtbd0p4
   Architecture: amd64
   Architecture Version: 2
   Dump Length: 893067264B (851 MB)
   Blocksize: 512
   Dumptime: Wed Mar  6 13:29:20 2013
   Hostname: dentredje.dvconsulting.dk
   Magic: FreeBSD Kernel Dump
   Version String: FreeBSD 9.1-RELEASE #0: Wed Mar  6 11:30:02 CET 2013
     root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC
   Panic String: page fault
   Dump Parity: 2869697314
   Bounds: 6
   Dump Status: good
 [root@dentredje crash]#=20
 
 core.txt.6
 
 ***
 [root@dentredje crash]# cat core.txt.6=20
 dentredje.dvconsulting.dk dumped core - see /var/crash/vmcore.6
 
 Wed Mar  6 13:32:32 CET 2013
 
 FreeBSD dentredje.dvconsulting.dk 9.1-RELEASE FreeBSD 9.1-RELEASE #0: =
 Wed Mar  6 11:30:02 CET 2013     =
 root@denfjerde.dvconsulting.dk:/usr/obj/usr/src/sys/GENERIC  amd64
 
 panic: page fault
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you =
 are
 welcome to change it and/or distribute copies of it under certain =
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for =
 details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 
 
 Fatal trap 12: page fault while in kernel mode
 cpuid =3D 0; apic id =3D 00
 fault virtual address	=3D 0x60
 fault code		=3D supervisor read data, page not present
 instruction pointer	=3D 0x20:0xffffffff8162e4f0
 stack pointer	        =3D 0x28:0xffffff8161b726e0
 frame pointer	        =3D 0x28:0xffffff8161b727d0
 code segment		=3D base 0x0, limit 0xfffff, type 0x1b
 			=3D DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags	=3D interrupt enabled, resume, IOPL =3D 0
 current process		=3D 2999 (zpool)
 trap number		=3D 12
 panic: page fault
 cpuid =3D 3
 KDB: stack backtrace:
 #0 0xffffffff809208a6 at kdb_backtrace+0x66
 #1 0xffffffff808ea8be at panic+0x1ce
 #2 0xffffffff80bd8240 at trap_fatal+0x290
 #3 0xffffffff80bd857d at trap_pfault+0x1ed
 #4 0xffffffff80bd8b9e at trap+0x3ce
 #5 0xffffffff80bc315f at calltrap+0x8
 #6 0xffffffff81673975 at sa_handle_get_from_db+0x95
 #7 0xffffffff81673a38 at sa_handle_get+0x48
 #8 0xffffffff8169f516 at zfs_grab_sa_handle+0x96
 #9 0xffffffff8169faca at zfs_obj_to_path+0x6a
 #10 0xffffffff816b8c75 at zfs_ioc_obj_to_path+0x75
 #11 0xffffffff816bad46 at zfsdev_ioctl+0xe6
 #12 0xffffffff807db28b at devfs_ioctl_f+0x7b
 #13 0xffffffff80932325 at kern_ioctl+0x115
 #14 0xffffffff8093255d at sys_ioctl+0xfd
 #15 0xffffffff80bd7ae6 at amd64_syscall+0x546
 #16 0xffffffff80bc3447 at Xfast_syscall+0xf7
 Uptime: 2m10s
 Dumping 851 out of 5100 =
 MB:..2%..12%..21%..31%..42%..51%..61%..72%..81%..91%
 
 Reading symbols from /boot/modules/virtio.ko...Reading symbols from =
 /boot/modules/virtio.ko.symbols...done.
 done.
 Loaded symbols for /boot/modules/virtio.ko
 Reading symbols from /boot/modules/virtio_pci.ko...Reading symbols from =
 /boot/modules/virtio_pci.ko.symbols...done.
 done.
 Loaded symbols for /boot/modules/virtio_pci.ko
 Reading symbols from /boot/modules/virtio_blk.ko...Reading symbols from =
 /boot/modules/virtio_blk.ko.symbols...done.
 done.
 Loaded symbols for /boot/modules/virtio_blk.ko
 Reading symbols from /boot/modules/if_vtnet.ko...Reading symbols from =
 /boot/modules/if_vtnet.ko.symbols...done.
 done.
 Loaded symbols for /boot/modules/if_vtnet.ko
 Reading symbols from /boot/modules/virtio_balloon.ko...Reading symbols =
 from /boot/modules/virtio_balloon.ko.symbols...done.
 done.
 Loaded symbols for /boot/modules/virtio_balloon.ko
 Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from =
 /boot/kernel/ipfw.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipfw.ko
 Reading symbols from /boot/kernel/ipdivert.ko...Reading symbols from =
 /boot/kernel/ipdivert.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipdivert.ko
 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from =
 /boot/kernel/zfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/zfs.ko
 Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from =
 /boot/kernel/opensolaris.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/opensolaris.ko
 #0  doadump (textdump=3DVariable "textdump" is not available.
 ) at pcpu.h:224
 224	pcpu.h: No such file or directory.
 	in pcpu.h
 (kgdb) #0  doadump (textdump=3DVariable "textdump" is not available.
 ) at pcpu.h:224
 #1  0xffffffff808ea3a1 in kern_reboot (howto=3D260)
     at /usr/src/sys/kern/kern_shutdown.c:448
 #2  0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
     at /usr/src/sys/kern/kern_shutdown.c:636
 #3  0xffffffff80bd8240 in trap_fatal (frame=3D0xc, eva=3DVariable "eva" =
 is not available.
 )
     at /usr/src/sys/amd64/amd64/trap.c:857
 #4  0xffffffff80bd857d in trap_pfault (frame=3D0xffffff8161b72630, =
 usermode=3D0)
     at /usr/src/sys/amd64/amd64/trap.c:773
 #5  0xffffffff80bd8b9e in trap (frame=3D0xffffff8161b72630)
     at /usr/src/sys/amd64/amd64/trap.c:456
 #6  0xffffffff80bc315f in calltrap ()
     at /usr/src/sys/amd64/amd64/exception.S:228
 #7  0xffffffff8162e4f0 in list_remove_tail ()
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/list=
 .c:151
 #8  0xffffffff816736af in sa_build_index (hdl=3D0xfffffe010ac98690,=20
     buftype=3DSA_BONUS)
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 sa.c:1490
 #9  0xffffffff81673975 in sa_handle_get_from_db (os=3D0xfffffe0132baf000,=20=
 
     db=3D0xfffffe010ac968c0, userp=3D0x0, hdl_type=3DVariable "hdl_type" =
 is not available.
 )
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 sa.c:1367
 #10 0xffffffff81673a38 in sa_handle_get (objset=3D0xfffffe0132baf000, =
 objid=3DVariable "objid" is not available.
 )
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 sa.c:1392
 #11 0xffffffff8169f516 in zfs_grab_sa_handle (osp=3D0xfffffe0132baf000,=20=
 
     obj=3D230809, hdlp=3D0xffffff8161b72908, db=3D0xffffff8161b72900,=20
     tag=3D0xffffffff81722e10)
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 zfs_znode.c:2031
 #12 0xffffffff8169faca in zfs_obj_to_path (osp=3D0xfffffe0132baf000,=20
     obj=3D230809, buf=3D0xffffff8008757400 "data2/src", len=3D2048)
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 zfs_znode.c:2184
 #13 0xffffffff816b8c75 in zfs_ioc_obj_to_path (zc=3D0xffffff8008757000)
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 zfs_ioctl.c:1537
 #14 0xffffffff816bad46 in zfsdev_ioctl (dev=3DVariable "dev" is not =
 available.
 )
     at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 zfs_ioctl.c:5218
 #15 0xffffffff807db28b in devfs_ioctl_f (fp=3D0xfffffe006bdf8690,=20
     com=3D3583531558, data=3DVariable "data" is not available.
 ) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
 #16 0xffffffff80932325 in kern_ioctl (td=3DVariable "td" is not =
 available.
 ) at file.h:293
 #17 0xffffffff8093255d in sys_ioctl (td=3D0xfffffe00033e58e0,=20
     uap=3D0xffffff8161b72bb0) at /usr/src/sys/kern/sys_generic.c:691
 #18 0xffffffff80bd7ae6 in amd64_syscall (td=3D0xfffffe00033e58e0, =
 traced=3D0)
     at subr_syscall.c:135
 #19 0xffffffff80bc3447 in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:387
 #20 0x0000000801debcdc in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 (kgdb)=20
 
 ------------------------------------------------------------------------
 ps -axl
 
 ps: can't read sigacts at 0xfffffe00a1c64000
 
 ------------------------------------------------------------------------
 vmstat -s
 
    392168 cpu context switches
     44454 device interrupts
      8725 software interrupts
   1299960 traps
   1763303 system calls
        19 kernel threads created
      2839  fork() calls
       141 vfork() calls
         0 rfork() calls
         0 swap pager pageins
         0 swap pager pages paged in
         0 swap pager pageouts
         0 swap pager pages paged out
      4580 vnode pager pageins
     19695 vnode pager pages paged in
         0 vnode pager pageouts
         0 vnode pager pages paged out
         0 page daemon wakeups
         0 pages examined by the page daemon
       426 pages reactivated
    206213 copy-on-write faults
       470 copy-on-write optimized faults
    947712 zero fill pages zeroed
         0 zero fill pages prezeroed
        86 intransit blocking page faults
   1274475 total VM faults taken
         0 pages affected by kernel thread creation
   1413892 pages affected by  fork()
     70812 pages affected by vfork()
         0 pages affected by rfork()
         0 pages cached
   1157439 pages freed
         0 pages freed by daemon
         0 pages freed by exiting processes
    107649 pages active
    203197 pages inactive
        59 pages in VM cache
    180298 pages wired down
    771791 pages free
      4096 bytes per page
    990965 total name lookups
           cache hits (89% pos + 5% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%
 
 ------------------------------------------------------------------------
 vmstat -m
 
          Type InUse MemUse HighUse Requests  Size(s)
 CAM dev queue     4     1K       -        4  128
   md_sii_data     0     0K       -       42  512
       CAM XPT    29    16K       -       95  32,64,128,1024,2048
        isadev     9     2K       -        9  128
      acpiintr     1     1K       -        1  64
          cdev     8     2K       -        8  256
        acpica  1707   188K       -    21034  =
 16,32,64,128,256,512,1024,2048
         sigio     1     1K       -        1  64
      filedesc   234   963K       -     3180  16,32,64,512,1024,2048,4096
          kenv    66    11K       -       70  16,32,64,128
        kqueue    24    27K       -      250  256,512,2048
     proc-args   141    13K       -     1766  16,32,64,128,256
         hhook     2     1K       -        2  128
      acpitask     1     2K       -        1  2048
       ithread    74    12K       -       74  32,128,256
     CAM queue    12     1K       -       71  16,32
        KTRACE   100    13K       -      100  128
        linker   261   897K       -      303  =
 16,32,64,128,256,512,1024,2048,4096
         lockf   148    15K       -     1787  64,128,256
    loginclass     2     1K       -        6  64
        ip6ndp    10     1K       -       13  64,128
          temp    20    13K       -    14346  =
 16,32,64,128,256,512,1024,2048,4096
        devbuf  3564 27445K       -     3609  =
 16,32,64,128,256,512,1024,2048,4096
        module   489    62K       -      489  128
      mtx_pool     2    16K       -        2 =20
        USBdev     4     1K       -        4  64,128
           osd     3     1K       -       16  16,64
           USB     7     3K       -        7  16,32,128,2048
      pmchooks     1     1K       -        1  128
       subproc   410   835K       -     3230  512,4096
          proc     2    16K       -        2 =20
       session    28     4K       -      143  128
          pgrp    38     5K       -      219  128
          cred   170    27K       -    58670  64,256
       uidinfo    24     5K       -      257  128,2048
        plimit    21     6K       -      201  256
       ata_pci     1     1K       -        1  64
     sysctltmp     0     0K       -      890  16,32,64,128,4096
     sysctloid  2923   145K       -     2998  16,32,64,128
        sysctl     0     0K       -      840  16,32,64
       tidhash     1    16K       -        1 =20
       callout     3  1536K       -        3 =20
          umtx   948   119K       -      948  128
      p1003.1b     1     1K       -        1  16
          SWAP     2   549K       -        2  64
        bus-sc    46   105K       -     2000  =
 16,32,128,256,512,1024,2048,4096
           bus  1036    83K       -     3883  16,32,64,128,256,1024
       devstat    16    33K       -       16  32,4096
  eventhandler    90     8K       -       90  64,128
       acpisem    16     2K       -       16  128
          kobj   334  1336K       -      692  4096
       CAM SIM     4     1K       -        4  256
       Per-cpu     1     1K       -        1  32
    CAM periph     2     1K       -       18  16,32,64,128,256
          rman   100    12K       -      508  16,32,128
          sbuf     0     0K       -     2357  =
 16,32,64,128,256,512,1024,2048,4096
        sglist     4     4K       -        4  1024
       entropy  1024    64K       -     1024  64
        ctlmem  5062 10113K       -     5062  128,2048
         stack     0     0K       -        2  256
     taskqueue    61     6K       -       91  16,32,64,128,1024
        Unitno    16     1K       -     2152  32,64
           iov     0     0K       -     4365  16,64,128,256,512
        select   195    25K       -      195  128
      ioctlops     1     8K       -    15314  =
 16,32,64,128,256,512,1024,2048,4096
           msg     4    30K       -        4  2048,4096
           sem     4   106K       -        4  2048,4096
           shm     1    20K       -        3  2048
           tty    20    20K       -       22  1024,2048
           pts     1     1K       -        1  256
      mbuf_tag     0     0K       -        7  32
         shmfd     1     8K       -        1 =20
           pcb    52   158K       -      809  16,32,128,1024,2048,4096
        soname    11     2K       -     7276  16,32,128
      vfscache     1  2048K       -        1 =20
    cl_savebuf     0     0K       -      310  64
      vfs_hash     1  1024K       -        1 =20
        DEVFS1    80    40K       -      102  512
        vnodes    28     3K       -      108  64,256
        DEVFS3   103    26K       -      132  256
         mount    86     4K       -      169  16,32,64,128,256
   vnodemarker     0     0K       -      120  512
           BPF     6     1K       -        6  128
   ether_multi    79     5K       -       94  16,32,64
        ifaddr    80    21K       -       80  32,64,128,256,512,4096
         ifnet     7    13K       -        7  128,2048
         clone     6    24K       -        6  4096
        arpcom     4     1K       -        4  16
       lltable    26    10K       -       26  256,512
         DEVFS    11     1K       -       12  16,128
      routetbl    56     7K       -      221  32,64,128,256,512
          igmp     6     2K       -        6  256
      in_multi     4     1K       -        4  256
     sctp_iter     0     0K       -       12  256
      sctp_ifn     4     1K       -        4  128
      sctp_ifa    14     2K       -       14  128
      sctp_vrf     1     1K       -        1  64
     sctp_a_it     0     0K       -       12  16
     hostcache     1    28K       -        1 =20
      syncache     1    96K       -        1 =20
     in6_multi    36     5K       -       36  32,256
           mld     6     1K       -        6  128
           rpc     2     1K       -        2  256
 audit_evclass   179     6K       -      218  32
       jblocks     6     2K       -        6  128,256
      savedino     0     0K       -      129  256
         sbdep     0     0K       -       17  64
       jsegdep   106     7K       -    25650  64
          jseg   270    34K       -      315  128
     jfreefrag     0     0K       -      829  128
       jnewblk     0     0K       -    23303  128
       jremref     8     1K       -      735  128
       jaddref     0     0K       -      783  128
       freedep     0     0K       -        4  64
      freework    21     3K       -      583  64,128
     newdirblk     0     0K       -       36  64
        dirrem    46     6K       -      663  128
         mkdir     0     0K       -       72  128
        diradd     1     1K       -      711  128
      freefile     9     1K       -      411  64
      freeblks    17     3K       -      438  128
      freefrag     0     0K       -      829  128
      indirdep     0     0K       -       12  128
        newblk     5   129K       -    23304  256
     bmsafemap     4     9K       -      654  256
      inodedep   113  1080K       -      919  512
       pagedep    52   141K       -      401  256
   ufs_dirhash    93    20K       -       94  16,32,64,128,256,512
     ufs_mount    15    67K       -       15  512,4096
     vm_pgdata     2   129K       -        2  128
       UMAHash     2     1K       -        2  512
       memdesc     1     4K       -        1  4096
      atkbddev     2     1K       -        2  64
     pfs_nodes    21     6K       -       21  256
        ctlblk   200  1600K       -      200 =20
          GEOM   127    26K       -     1541  =
 16,32,64,128,256,512,1024,2048
       ramdisk     1  4096K       -        1 =20
       acpidev    18     2K       -       18  64
       ctlpool   532   142K       -      532  32,512
        kbdmux     6    18K       -        6  16,512,1024,2048
           LED     4     1K       -        4  16,128
        apmdev     1     1K       -        1  128
    madt_table     0     0K       -        1  4096
        feeder     7     1K       -        7  32
      pci_link    10     1K       -       10  16,128
     raid_data     0     0K       -      252  32,128,256
       io_apic     1     2K       -        1  2048
           MCA    10     2K       -       10  128
      nexusdev     3     1K       -        3  16
 md_nvidia_data     0     0K       -       42  512
   IpFw/IpAcct    20     5K       -       35  16,32,64,128,1024
       solaris 42732 117071K       -   516982  =
 16,32,64,128,256,512,1024,2048,4096
    kstat_data     4     1K       -        4  64
 
 ------------------------------------------------------------------------
 vmstat -z
 
 ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
 
 UMA Kegs:               208,      0,     188,      16,     188,   0,   0
 UMA Zones:              896,      0,     188,       0,     188,   0,   0
 UMA Slabs:              568,      0,   10842,       1,   17497,   0,   0
 UMA RCntSlabs:          568,      0,     811,       1,     811,   0,   0
 UMA Hash:               256,      0,      79,      11,      81,   0,   0
 16 Bucket:              152,      0,     140,      10,     140,   0,   0
 32 Bucket:              280,      0,     119,       7,     119,   2,   0
 64 Bucket:              536,      0,     125,       1,     125,  57,   0
 128 Bucket:            1048,      0,     492,       0,     492,1050,   0
 VM OBJECT:              232,      0,    8228,     204,   60846,   0,   0
 MAP:                    232,      0,       7,      25,       7,   0,   0
 KMAP ENTRY:             120, 182559,      99,     521,   22390,   0,   0
 MAP ENTRY:              120,      0,    8567,     330,  147448,   0,   0
 fakepg:                 120,      0,       0,       0,       0,   0,   0
 mt_zone:               4112,      0,     331,      42,     331,   0,   0
 16:                      16,      0,    3858,     342,   53852,   0,   0
 32:                      32,      0,    4201,     647,   46199,   0,   0
 64:                      64,      0,   23763,   15269,  271306,   0,   0
 128:                    128,      0,    6587,     431,  130489,   0,   0
 256:                    256,      0,    6884,   15661,  117624,   0,   0
 512:                    512,      0,    4884,     128,   76796,   0,   0
 1024:                  1024,      0,     845,    1207,    7696,   0,   0
 2048:                  2048,      0,    5580,     330,    6845,   0,   0
 4096:                  4096,      0,    1816,     131,   39969,   0,   0
 Files:                   80,      0,     969,     246,   48815,   0,   0
 TURNSTILE:              136,      0,     475,      85,     475,   0,   0
 umtx pi:                 96,      0,       0,       0,       0,   0,   0
 MAC labels:              40,      0,       0,       0,       0,   0,   0
 PROC:                  1184,      0,     179,      49,    2999,   0,   0
 THREAD:                1128,      0,     429,      45,     591,   0,   0
 SLEEPQUEUE:              80,      0,     475,      76,     475,   0,   0
 VMSPACE:                392,      0,     161,      49,    2985,   0,   0
 cpuset:                  72,      0,     206,     194,     348,   0,   0
 audit_record:           960,      0,       0,       0,       0,   0,   0
 mbuf_packet:            256,      0,     602,     813,   32072,   0,   0
 mbuf:                   256,      0,      14,     633,   36538,   0,   0
 mbuf_cluster:          2048,  25600,    1408,      60,    1408,   0,   0
 mbuf_jumbo_page:       4096,  12800,       0,      77,    1138,   0,   0
 mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   0
 mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   0
 mbuf_ext_refcnt:          4,      0,       0,     672,     185,   0,   0
 g_bio:                  232,      0,       0,     640,   95668,   0,   0
 ttyinq:                 160,      0,     135,      81,     270,   0,   0
 ttyoutq:                256,      0,      72,      48,     144,   0,   0
 ata_request:            328,      0,       0,       0,       0,   0,   0
 ata_composite:          336,      0,       0,       0,       0,   0,   0
 vtblk_request:           48,      0,     512,      64,     512,   0,   0
 vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
 VNODE:                  480,      0,    6185,      87,    7601,   0,   0
 VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
 S VFS Cache:            108,      0,    5866,     140,   46996,   0,   0
 STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
 L VFS Cache:            328,      0,     411,     177,    3463,   0,   0
 LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
 NAMEI:                 1024,      0,       1,      67,  210673,   0,   0
 NCLNODE:                568,      0,       0,       0,       0,   0,   0
 DIRHASH:               1024,      0,     206,      34,     206,   0,   0
 Mountpoints:            792,      0,       7,       8,       7,   0,   0
 pipe:                   728,      0,      53,      52,    2023,   0,   0
 ksiginfo:               112,      0,     214,     842,     290,   0,   0
 itimer:                 344,      0,       1,      21,       1,   0,   0
 KNOTE:                  128,      0,      69,     105,     342,   0,   0
 socket:                 680,  25602,     289,      65,    2277,   0,   0
 unpcb:                  240,  25600,     164,      92,     687,   0,   0
 ipq:                     56,    819,       0,       0,       0,   0,   0
 udp_inpcb:              392,  25600,      29,      71,    1126,   0,   0
 udpcb:                   16,  25704,      29,     643,    1126,   0,   0
 tcp_inpcb:              392,  25600,     120,      60,     440,   0,   0
 tcpcb:                  976,  25600,      85,      27,     440,   0,   0
 tcptw:                   72,   5150,      35,     215,      37,   0,   0
 syncache:               152,  15375,       1,     124,     186,   0,   0
 hostcache:              136,  15372,      57,     111,      57,   0,   0
 tcpreass:                40,   1680,      17,     403,     398,   0,   0
 sackhole:                32,      0,       0,     303,      26,   0,   0
 sctp_ep:               1376,  25600,       0,       0,       0,   0,   0
 sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
 sctp_laddr:              48,  80064,       0,     288,      13,   0,   0
 sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
 sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
 sctp_readq:             104, 400032,       0,       0,       0,   0,   0
 sctp_stream_msg_out:    112, 400026,       0,       0,       0,   0,   0
 sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
 sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
 ripcb:                  392,  25600,       0,      50,      18,   0,   0
 rtentry:                200,      0,      34,      80,      34,   0,   0
 selfd:                   56,      0,     432,     387,   92002,   0,   0
 SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
 FFS inode:              168,      0,    3499,     109,    3934,   0,   0
 FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
 FFS2 dinode:            256,      0,    3499,     161,    3926,   0,   0
 IPFW dynamic rule:      120,      0,       0,       0,       0,   0,   0
 divcb:                  392,  25600,       0,       0,       0,   0,   0
 taskq_zone:              48,      0,       0,     360,      64,   0,   0
 zio_cache:              880,      0,       1,    1599,   75260,   0,   0
 zio_link_cache:          48,      0,       0,    2448,   37923,   0,   0
 zio_buf_512:            512,      0,       0,       0,       0,   0,   0
 zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
 zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
 zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
 zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
 zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
 zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
 zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
 zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
 zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
 zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
 zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
 zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
 zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
 zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
 zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
 zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
 zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
 zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
 zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
 zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
 zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
 zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
 zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
 zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
 zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
 zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
 zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
 zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
 zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
 zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
 zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
 zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
 zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
 zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
 zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
 zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
 zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
 zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
 zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
 zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
 zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
 zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
 zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
 zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
 zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
 zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
 zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
 zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
 zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
 zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
 zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
 zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
 zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
 zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
 zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
 zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
 zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
 zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
 zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
 zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
 zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
 zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
 zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
 zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
 zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
 zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
 zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
 zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
 zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
 zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
 zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
 zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
 zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
 zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
 zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
 zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
 zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
 zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
 zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
 zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
 zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
 zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
 zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
 zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
 zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
 zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
 zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
 sa_cache:                80,      0,    2647,     143,    3636,   0,   0
 dnode_t:                856,      0,    3641,     111,    4026,   0,   0
 dmu_buf_impl_t:         224,      0,    7859,     250,   10606,   0,   0
 arc_buf_hdr_t:          216,      0,    6225,      93,    8090,   0,   0
 arc_buf_t:              104,      0,    6135,     165,    8127,   0,   0
 zil_lwb_cache:          192,      0,       1,      99,      23,   0,   0
 zfs_znode_cache:        400,      0,    2646,      45,    3625,   0,   0
 
 
 ------------------------------------------------------------------------
 vmstat -i
 
 interrupt                          total       rate
 irq1: atkbd0                           8          0
 irq10: em0 em1+                    21867        173
 irq11: re0 uhci0++*                22579        179
 cpu0:timer                         43190        342
 cpu3:timer                         31518        250
 cpu2:timer                         35619        282
 cpu1:timer                         44648        354
 Total                             199429       1582
 
 ------------------------------------------------------------------------
 pstat -T
 
 969/12328 files
 0M/4095M swap space
 
 ------------------------------------------------------------------------
 pstat -s
 
 Device          512-blocks     Used    Avail Capacity
 /dev/vtbd0p4       8388352        0  8388352     0%
 
 ------------------------------------------------------------------------
 iostat
 
 iostat: kvm_read(_tk_nin): invalid address (0x0)
 iostat: disabling TTY statistics
            vtbd0            vtbd1            vtbd2             cpu
   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  83.94 168 13.73   7.24  39  0.28  11.92  30  0.35  13  0  7  1 79
 
 ------------------------------------------------------------------------
 ipcs -a
 
 Message Queues:
 T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
 CGROUP                 CBYTES                 QNUM               QBYTES  =
       LSPID        LRPID STIME    RTIME    CTIME  =20
 
 Shared Memory:
 T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
 CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    =
 DTIME    CTIME  =20
 m        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   munin =
               0        65536         1601         2199 13:28:39 13:28:39 =
 13:27:56
 
 Semaphores:
 T           ID          KEY MODE        OWNER    GROUP    CREATOR  =
 CGROUP          NSEMS OTIME    CTIME  =20
 s        65536   1667461225 --rw-rw-rw- nobody   munin    nobody   munin =
               3 13:28:39 13:27:56
 
 
 ------------------------------------------------------------------------
 ipcs -T
 
 msginfo:
 	msgmax:        16384	(max characters in a message)
 	msgmni:           40	(# of message queues)
 	msgmnb:         2048	(max characters in a message queue)
 	msgtql:           40	(max # of messages in system)
 	msgssz:            8	(size of a message segment)
 	msgseg:         2048	(# of message segments in system)
 
 shminfo:
 	shmmax:    536870912	(max shared memory segment size)
 	shmmin:            1	(min shared memory segment size)
 	shmmni:          192	(max number of shared memory =
 identifiers)
 	shmseg:          128	(max shared memory segments per process)
 	shmall:       131072	(max amount of shared memory in pages)
 
 seminfo:
 	semmni:           50	(# of semaphore identifiers)
 	semmns:          340	(# of semaphores in system)
 	semmnu:          150	(# of undo structures in system)
 	semmsl:          340	(max # of semaphores per id)
 	semopm:          100	(max # of operations per semop call)
 	semume:           50	(max # of undo entries per process)
 	semusz:          632	(size in bytes of undo structure)
 	semvmx:        32767	(semaphore maximum value)
 	semaem:        16384	(adjust on exit max value)
 
 
 ------------------------------------------------------------------------
 nfsstat
 
 Client Info:
 Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    =
 Remove
         0         0         0         0         0         0         0    =
      0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    =
 Access
         0         0         0         0         0         0         0    =
      0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
 Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0         0
 Cache Info:
 Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    =
 Misses
         0         0         0         0         0         0         0    =
      0
 BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    =
 Misses
         0         0         0         0         0         0         0    =
      0
 
 Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    =
 Remove
         0         0         0         0         0         0         0    =
      0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    =
 Access
         0         0         0         0         0         0         0    =
      0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
 Server Ret-Failed
                 0
 Server Faults
             0
 Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0         0
 Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
         0         0         0
 
 ------------------------------------------------------------------------
 netstat -s
 
 tcp:
 	9385 packets sent
 		6192 data packets (6331579 bytes)
 		99 data packets (127875 bytes) retransmitted
 		4 data packets unnecessarily retransmitted
 		0 resends initiated by MTU discovery
 		2119 ack-only packets (214 delayed)
 		0 URG only packets
 		0 window probe packets
 		768 window update packets
 		209 control packets
 	10672 packets received
 		4817 acks (for 6305478 bytes)
 		222 duplicate acks
 		0 acks for unsent data
 		4579 packets (4011692 bytes) received in-sequence
 		34 completely duplicate packets (43596 bytes)
 		0 old duplicate packets
 		0 packets with some dup. data (0 bytes duped)
 		398 out-of-order packets (533435 bytes)
 		0 packets (0 bytes) of data after window
 		0 window probes
 		91 window update packets
 		78 packets received after close
 		0 discarded for bad checksums
 		0 discarded for bad header offset fields
 		0 discarded because packet too short
 		12 discarded due to memory problems
 	86 connection requests
 	185 connection accepts
 	0 bad connection attempts
 	0 listen queue overflows
 	8 ignored RSTs in the windows
 	245 connections established (including accepts)
 	321 connections closed (including 40 drops)
 		153 connections updated cached RTT on close
 		153 connections updated cached RTT variance on close
 		7 connections updated cached ssthresh on close
 	26 embryonic connections dropped
 	4733 segments updated rtt (of 3794 attempts)
 	27 retransmit timeouts
 		0 connections dropped by rexmit timeout
 	0 persist timeouts
 		0 connections dropped by persist timeout
 	0 Connections (fin_wait_2) dropped because of timeout
 	0 keepalive timeouts
 		0 keepalive probes sent
 		0 connections dropped by keepalive
 	359 correct ACK header predictions
 	2854 correct data packet header predictions
 	186 syncache entries added
 		0 retransmitted
 		1 dupsyn
 		0 dropped
 		185 completed
 		0 bucket overflow
 		0 cache overflow
 		0 reset
 		0 stale
 		0 aborted
 		0 badack
 		0 unreach
 		0 zone failures
 	186 cookies sent
 	0 cookies received
 	57 hostcache entries added
 		0 bucket overflow
 	3 SACK recovery episodes
 	36 segment rexmits in SACK recovery episodes
 	52128 byte rexmits in SACK recovery episodes
 	92 SACK options (SACK blocks) received
 	503 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 packets with ECN CE bit set
 	0 packets with ECN ECT(0) bit set
 	0 packets with ECN ECT(1) bit set
 	0 successful ECN handshakes
 	0 times ECN reduced the congestion window
 udp:
 	3761 datagrams received
 	0 with incomplete header
 	0 with bad data length field
 	0 with bad checksum
 	1 with no checksum
 	1279 dropped due to no socket
 	0 broadcast/multicast datagrams undelivered
 	0 dropped due to full socket buffers
 	0 not for hashed pcb
 	2482 delivered
 	2483 datagrams output
 	0 times multicast source filter matched
 ip:
 	14453 total packets received
 	0 bad header checksums
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with ip length > max ip packet size
 	0 with header length < data size
 	0 with data length < header length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 packets reassembled ok
 	14435 packets for this host
 	12 packets for unknown/unsupported protocol
 	0 packets forwarded (0 packets fast forwarded)
 	0 packets not forwardable
 	0 packets received for unknown multicast group
 	0 redirects sent
 	13495 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	387 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 tunneling packets that can't find gif
 	0 datagrams with bad address in header
 icmp:
 	746 calls to icmp_error
 	0 errors not generated in response to an icmp message
 	Output histogram:
 		echo reply: 2
 		destination unreachable: 746
 	0 messages with bad code fields
 	0 messages less than the minimum length
 	0 messages with bad checksum
 	0 messages with bad length
 	0 multicast echo requests ignored
 	0 multicast timestamp requests ignored
 	Input histogram:
 		destination unreachable: 7
 		echo: 2
 	2 message responses generated
 	0 invalid return addresses
 	0 no return routes
 igmp:
 	5 messages received
 	0 messages received with too few bytes
 	0 messages received with wrong TTL
 	0 messages received with bad checksum
 	5 V1/V2 membership queries received
 	0 V3 membership queries received
 	0 membership queries received with invalid field(s)
 	5 general queries received
 	0 group queries received
 	0 group-source queries received
 	0 group-source queries dropped
 	0 membership reports received
 	0 membership reports received with invalid field(s)
 	0 membership reports received for groups to which we belong
 	0 V3 reports received without Router Alert
 	0 membership reports sent
 arp:
 	9 ARP requests sent
 	0 ARP replies sent
 	66 ARP requests received
 	1 ARP reply received
 	67 ARP packets received
 	0 total packets dropped due to no ARP entry
 	0 ARP entrys timed out
 	0 Duplicate IPs seen
 ip6:
 	0 total packets received
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 fragments that exceeded limit
 	0 packets reassembled ok
 	0 packets for this host
 	0 packets forwarded
 	0 packets not forwardable
 	0 redirects sent
 	11 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	0 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 packets that violated scope rules
 	0 multicast packets which we don't join
 	Mbuf statistics:
 		0 one mbuf
 		3 one ext mbuf
 		0 two or more ext mbuf
 	0 packets whose headers are not contiguous
 	0 tunneling packets that can't find gif
 	0 packets discarded because of too many headers
 	0 failures of source address selection
 	Source addresses selection rule applied:
 		16 same address
 icmp6:
 	0 calls to icmp6_error
 	0 errors not generated in response to an icmp6 message
 	0 errors not generated because of rate limitation
 	Output histogram:
 		neighbor solicitation: 3
 	0 messages with bad code fields
 	0 messages < minimum length
 	0 bad checksums
 	0 messages with bad length
 	Histogram of error messages to be generated:
 		0 no route
 		0 administratively prohibited
 		0 beyond scope
 		0 address unreachable
 		0 port unreachable
 		0 packet too big
 		0 time exceed transit
 		0 time exceed reassembly
 		0 erroneous header field
 		0 unrecognized next header
 		0 unrecognized option
 		0 redirect
 		0 unknown
 	0 message responses generated
 	0 messages with too many ND options
 	0 messages with bad ND options
 	0 bad neighbor solicitation messages
 	0 bad neighbor advertisement messages
 	0 bad router solicitation messages
 	0 bad router advertisement messages
 	0 bad redirect messages
 	0 path MTU changes
 rip6:
 	0 messages received
 	0 checksum calculations on inbound
 	0 messages with bad checksum
 	0 messages dropped due to no socket
 	0 multicast messages dropped due to no socket
 	0 messages dropped due to full socket buffers
 	0 delivered
 	0 datagrams output
 
 ------------------------------------------------------------------------
 netstat -m
 
 616/1446/2062 mbufs in use (current/cache/total)
 595/873/1468/25600 mbuf clusters in use (current/cache/total/max)
 602/813 mbuf+clusters out of packet secondary zone in use =
 (current/cache)
 0/77/77/12800 4k (page size) jumbo clusters in use =
 (current/cache/total/max)
 0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
 1344K/2415K/3759K bytes allocated to network (current/cache/total)
 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 59 requests for I/O initiated by sendfile
 0 calls to protocol drain routines
 
 ------------------------------------------------------------------------
 netstat -id
 
 Name    Mtu Network       Address              Ipkts Ierrs Idrop    =
 Opkts Oerrs  Coll Drop
 usbus     0 <Link#1>                               0     0     0        =
 0     0     0    0=20
 re0    1500 <Link#2>      00:16:8e:1a:f3:a9        2     0     0        =
 1     0     0    0=20
 re0    1500 192.168.122.0 192.168.122.3            0     -     -        =
 0     -     -    -=20
 re0    1500 fe80::216:8ef fe80::216:8eff:fe        0     -     -        =
 2     -     -    -=20
 em0    1500 <Link#3>      52:54:00:b2:12:8c    13686     0     0    =
 12271     0     0    0=20
 em0    1500 195.234.155.0 dentredje              921     -     -    =
 12265     -     -    -=20
 em0    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -        =
 1     -     -    -=20
 em0    1500 195.234.155.0 skaarup.org           9594     -     -        =
 0     -     -    -=20
 em0    1500 195.234.155.0 ns11.skaarupaps.d        6     -     -        =
 0     -     -    -=20
 em0    1500 195.234.155.0 195.234.155.50           0     -     -        =
 0     -     -    -=20
 em0    1500 195.234.155.0 ntp.gal.dk            3079     -     -        =
 0     -     -    -=20
 em0    1500 195.234.155.0 195.234.155.75           0     -     -        =
 0     -     -    -=20
 em1    1500 <Link#4>      52:54:00:b2:12:a2       15     0     0        =
 1     0     0    0=20
 em1    1500 10.0.0.0      dentredjeb               0     -     -        =
 0     -     -    -=20
 em1    1500 fe80::5054:ff fe80::5054:ff:feb        0     -     -        =
 1     -     -    -=20
 lo0   16384 <Link#5>                             842     0     0      =
 842     0     0    0=20
 lo0   16384 localhost     ::1                      0     -     -        =
 0     -     -    -=20
 lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        =
 0     -     -    -=20
 lo0   16384 your-net      localhost              842     -     -      =
 842     -     -    -=20
 ipfw0 65536 <Link#6>                               0     0     0        =
 0     0     0    0=20
 
 ------------------------------------------------------------------------
 netstat -anr
 
 Routing tables
 
 Internet:
 Destination        Gateway            Flags    Refs      Use  Netif =
 Expire
 default            195.234.155.1      UGS         1    12267    em0
 10.0.0.0/24        link#4             U           0        0    em1
 10.0.0.3           link#4             UHS         0        0    lo0
 127.0.0.1          link#5             UH          0      842    lo0
 192.168.122.0/24   link#2             U           0        0    re0
 192.168.122.3      link#2             UHS         0        0    lo0
 195.234.155.0/24   link#3             U           0        0    em0
 195.234.155.50     link#3             UHS         0        0    lo0
 195.234.155.70     link#3             UHS         0        0    lo0
 195.234.155.75     link#3             UHS         0        0    lo0
 195.234.155.102    link#3             UHS         0        0    lo0
 195.234.155.124    link#3             UHS         0        0    lo0
 195.234.155.143    link#3             UHS         0        0    lo0
 
 Internet6:
 Destination                       Gateway                       Flags    =
   Netif Expire
 ::/96                             ::1                           UGRS     =
    lo0
 ::1                               link#5                        UH       =
    lo0
 ::ffff:0.0.0.0/96                 ::1                           UGRS     =
    lo0
 fe80::/10                         ::1                           UGRS     =
    lo0
 fe80::%re0/64                     link#2                        U        =
    re0
 fe80::216:8eff:fe1a:f3a9%re0      link#2                        UHS      =
    lo0
 fe80::%em0/64                     link#3                        U        =
    em0
 fe80::5054:ff:feb2:128c%em0       link#3                        UHS      =
    lo0
 fe80::%em1/64                     link#4                        U        =
    em1
 fe80::5054:ff:feb2:12a2%em1       link#4                        UHS      =
    lo0
 fe80::%lo0/64                     link#5                        U        =
    lo0
 fe80::1%lo0                       link#5                        UHS      =
    lo0
 ff01::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U        =
    re0
 ff01::%em0/32                     fe80::5054:ff:feb2:128c%em0   U        =
    em0
 ff01::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U        =
    em1
 ff01::%lo0/32                     ::1                           U        =
    lo0
 ff02::/16                         ::1                           UGRS     =
    lo0
 ff02::%re0/32                     fe80::216:8eff:fe1a:f3a9%re0  U        =
    re0
 ff02::%em0/32                     fe80::5054:ff:feb2:128c%em0   U        =
    em0
 ff02::%em1/32                     fe80::5054:ff:feb2:12a2%em1   U        =
    em1
 ff02::%lo0/32                     ::1                           U        =
    lo0
 
 ------------------------------------------------------------------------
 netstat -anA
 
 Active Internet connections (including servers)
 Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address  =
   (state)
 fffffe010a292000 tcp4       0      0 195.234.155.143.20 =
 208.83.137.114.270 ESTABLISHED
 fffffe010a28c7a0 tcp4       0      0 195.234.155.143.25 =
 81.19.253.131.6340 CLOSE_WAIT
 fffffe00a1069000 tcp4       0      0 195.234.155.143.25 =
 208.83.137.117.270 CLOSED
 fffffe00a106a3d0 tcp4       0      0 195.234.155.70.25  =
 81.30.22.111.11067 ESTABLISHED
 fffffe01326973d0 tcp4       0      6 195.234.155.143.43 173.194.69.26.25 =
   FIN_WAIT_1
 fffffe010a2933d0 tcp4       0      0 195.234.155.143.50 78.156.102.15.25 =
   ESTABLISHED
 fffffe00a106a000 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.387 ESTABLISHED
 fffffe013219a7a0 tcp4       0      0 127.0.0.1.26       127.0.0.1.11423  =
   ESTABLISHED
 fffffe006b41ab70 tcp4       0      0 127.0.0.1.11423    127.0.0.1.26     =
   ESTABLISHED
 fffffe00a10693d0 tcp4       0      0 127.0.0.1.26       127.0.0.1.32356  =
   ESTABLISHED
 fffffe010a2927a0 tcp4       0      0 127.0.0.1.32356    127.0.0.1.26     =
   ESTABLISHED
 fffffe01326977a0 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.329 ESTABLISHED
 fffffe010a292b70 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.394 ESTABLISHED
 fffffe00a10697a0 tcp4       0      0 195.234.155.143.26 81.19.253.131.25 =
   LAST_ACK
 fffffe00a1069b70 tcp4       0      0 195.234.155.70.143 =
 178.249.49.187.366 ESTABLISHED
 fffffe006bd383d0 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.450 ESTABLISHED
 fffffe013214d7a0 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.392 ESTABLISHED
 fffffe0006670000 tcp4    1448      0 195.234.155.70.993 =
 178.249.49.187.484 ESTABLISHED
 fffffe00a1d72b70 tcp4       0      0 195.234.155.70.993 =
 178.249.49.187.381 ESTABLISHED
 fffffe010a293000 tcp4       0      0 195.234.155.70.143 =
 178.249.49.187.400 ESTABLISHED
 fffffe01326d0000 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59864 ESTABLISHED
 fffffe010a28cb70 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59863 ESTABLISHED
 fffffe010a2937a0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59862 ESTABLISHED
 fffffe010a28d000 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59861 ESTABLISHED
 fffffe01327306c0 tcp4       0      0 195.234.155.70.25  =
 83.77.53.106.12850 TIME_WAIT
 fffffe010a28c3d0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59858 ESTABLISHED
 fffffe0132730708 tcp4       0      0 195.234.155.70.25  =
 85.49.162.228.5748 TIME_WAIT
 fffffe010a28d3d0 tcp4       0      0 195.234.155.70.25  =
 93.163.242.142.457 ESTABLISHED
 fffffe0132730750 tcp4       0      0 195.234.155.70.25  =
 78.131.218.227.329 TIME_WAIT
 fffffe0132730798 tcp4       0      0 195.234.155.70.25  =
 189.6.215.52.60453 TIME_WAIT
 fffffe01327307e0 tcp4       0      0 195.234.155.70.25  =
 77.229.19.23.55285 TIME_WAIT
 fffffe0132730828 tcp4       0      0 195.234.155.70.80  =
 88.208.217.154.452 TIME_WAIT
 fffffe00066707a0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62467 ESTABLISHED
 fffffe00a1d74000 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62466 ESTABLISHED
 fffffe00a1d743d0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62465 ESTABLISHED
 fffffe0132730870 tcp4       0      0 195.234.155.70.25  =
 91.39.55.104.31089 TIME_WAIT
 fffffe006bd373d0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62464 ESTABLISHED
 fffffe006bd393d0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62463 ESTABLISHED
 fffffe006bd387a0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62462 ESTABLISHED
 fffffe00a106c3d0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62461 ESTABLISHED
 fffffe00a106c7a0 tcp4       0      0 195.234.155.70.143 =
 80.162.6.187.62460 ESTABLISHED
 fffffe00a1d72000 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59837 FIN_WAIT_2
 fffffe00a1d723d0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59836 FIN_WAIT_2
 fffffe00a1d727a0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59835 FIN_WAIT_2
 fffffe00a1d73000 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59834 FIN_WAIT_2
 fffffe00a1d733d0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59833 FIN_WAIT_2
 fffffe00a1d737a0 tcp4       0      0 195.234.155.70.80  =
 193.88.216.6.59832 FIN_WAIT_2
 fffffe0132730a20 tcp4       0      0 195.234.155.70.25  =
 190.179.18.226.264 TIME_WAIT
 fffffe01327309d8 tcp4       0      0 195.234.155.70.25  =
 187.54.5.117.2375  TIME_WAIT
 fffffe010a473288 tcp4       0      0 195.234.155.70.80  =
 80.63.33.34.15139  TIME_WAIT
 fffffe01327308b8 tcp4       0      0 195.234.155.70.80  =
 80.63.33.34.15138  TIME_WAIT
 fffffe0132730900 tcp4       0      0 195.234.155.70.80  =
 80.63.33.34.15136  TIME_WAIT
 fffffe0132730990 tcp4       0      0 195.234.155.70.80  =
 80.63.33.34.15135  TIME_WAIT
 fffffe0132730948 tcp4       0      0 195.234.155.70.80  =
 80.63.33.34.15129  TIME_WAIT
 fffffe01326db7a0 tcp4       0      0 195.234.155.70.993 =
 5.103.173.126.1038 ESTABLISHED
 fffffe0132684000 tcp4       0      0 195.234.155.70.993 =
 5.103.173.126.1038 ESTABLISHED
 fffffe006bd377a0 tcp4       0      0 195.234.155.70.993 =
 5.103.173.126.1038 ESTABLISHED
 fffffe0132730a68 tcp4       0      0 195.234.155.70.25  =
 84.228.228.52.2942 TIME_WAIT
 fffffe01326da7a0 tcp4       0      0 195.234.155.70.993 =
 5.103.173.126.1038 ESTABLISHED
 fffffe01326d03d0 tcp4       0      0 195.234.155.70.993 =
 5.103.173.126.1037 LAST_ACK
 fffffe0132730af8 tcp4       0      0 195.234.155.70.25  =
 81.202.62.25.61332 TIME_WAIT
 fffffe0132730ab0 tcp4       0      0 195.234.155.70.25  =
 201.212.179.57.332 TIME_WAIT
 fffffe0132730b88 tcp4       0      0 195.234.155.70.80  =
 157.55.36.49.20092 TIME_WAIT
 fffffe0132730b40 tcp4       0      0 195.234.155.70.25  =
 81.34.117.66.50711 TIME_WAIT
 fffffe0132730bd0 tcp4       0      0 195.234.155.70.80  =
 157.55.35.76.14663 TIME_WAIT
 fffffe0132730c18 tcp4       0      0 195.234.155.70.80  =
 199.30.20.103.3187 TIME_WAIT
 fffffe0132730c60 tcp4       0      0 195.234.155.70.110 =
 188.183.53.110.607 TIME_WAIT
 fffffe0132730ca8 tcp4       0      0 195.234.155.70.80  =
 131.253.38.99.6317 TIME_WAIT
 fffffe006bd37b70 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.455 ESTABLISHED
 fffffe01326d1dc8 tcp4       0      0 195.234.155.70.25  79.42.1.16.21112 =
   TIME_WAIT
 fffffe0132730cf0 tcp4       0      0 195.234.155.70.25  =
 5.22.76.122.15314  TIME_WAIT
 fffffe006bd38b70 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.332 ESTABLISHED
 fffffe006bd39000 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.446 ESTABLISHED
 fffffe006bd397a0 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.432 ESTABLISHED
 fffffe006bd39b70 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.457 ESTABLISHED
 fffffe0132697000 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.448 ESTABLISHED
 fffffe0132684b70 tcp4       0      0 195.234.155.70.993 =
 87.72.76.248.50286 ESTABLISHED
 fffffe0132730d38 tcp4       0      0 195.234.155.70.25  =
 189.6.215.52.61730 TIME_WAIT
 fffffe010a473240 tcp4       0      0 195.234.155.70.25  =
 91.99.166.18.9811  TIME_WAIT
 fffffe013214d000 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.422 ESTABLISHED
 fffffe0006d95000 tcp4       0      0 127.0.0.1.26       *.*              =
   LISTEN
 fffffe01321993d0 tcp4       0      0 *.25               *.*              =
   LISTEN
 fffffe01326da000 tcp4       0      0 127.0.0.1.53       *.*              =
   LISTEN
 fffffe0006f7db70 tcp4       0      0 *.587              *.*              =
   LISTEN
 fffffe013219ab70 tcp4       0      0 127.0.0.1.3306     *.*              =
   LISTEN
 fffffe01321997a0 tcp4       0      0 195.234.155.70.25  =
 83.49.89.31.58370  FIN_WAIT_2
 fffffe00066703d0 tcp4       0      0 195.234.155.70.993 =
 85.233.252.158.374 ESTABLISHED
 fffffe0006670b70 tcp4       0      0 10.0.0.3.4949      *.*              =
   LISTEN
 fffffe01326dab70 tcp46      0      0 *.110              *.*              =
   LISTEN
 fffffe006b41a7a0 tcp46      0      0 *.993              *.*              =
   LISTEN
 fffffe013214db70 tcp46      0      0 *.143              *.*              =
   LISTEN
 fffffe006b469000 tcp4       0      0 195.234.155.70.80  =
 65.55.215.54.65193 TIME_WAIT
 fffffe0132730dc8 tcp4       0      0 195.234.155.70.25  =
 80.122.8.131.65026 TIME_WAIT
 fffffe0132730d80 tcp4       0      0 195.234.155.70.25  =
 190.179.62.166.249 TIME_WAIT
 fffffe01326d07a0 tcp4       0      0 *.443              *.*              =
   LISTEN
 fffffe01326d0b70 tcp6       0      0 *.443              *.*              =
   LISTEN
 fffffe006b419000 tcp4       0      0 *.82               *.*              =
   LISTEN
 fffffe006b4193d0 tcp6       0      0 *.82               *.*              =
   LISTEN
 fffffe006b4197a0 tcp4       0      0 *.80               *.*              =
   LISTEN
 fffffe006b419b70 tcp6       0      0 *.80               *.*              =
   LISTEN
 fffffe006b469048 tcp4       0      0 195.234.155.70.110 =
 85.81.181.86.49004 TIME_WAIT
 fffffe006b4690d8 tcp4       0      0 195.234.155.70.110 =
 85.81.181.86.38322 TIME_WAIT
 fffffe006b469090 tcp4       0      0 195.234.155.70.110 =
 85.81.181.86.47880 TIME_WAIT
 fffffe006b469168 tcp4       0      0 195.234.155.70.25  =
 190.138.186.235.19 TIME_WAIT
 fffffe006b469120 tcp4       0      0 195.234.155.70.25  =
 201.188.10.227.302 TIME_WAIT
 fffffe0132199000 tcp4       0     95 195.234.155.70.25  =
 212.156.16.66.2593 ESTABLISHED
 fffffe013219a000 tcp4       0      0 195.234.155.70.22  =
 85.233.252.158.373 ESTABLISHED
 fffffe01326db000 tcp4       0      0 *.21               *.*              =
   LISTEN
 fffffe013219a3d0 tcp4       0      0 127.0.0.1.51629    127.0.0.1.51635  =
   ESTABLISHED
 fffffe0006d953d0 tcp4       0      0 127.0.0.1.51635    127.0.0.1.51629  =
   ESTABLISHED
 fffffe01326db3d0 tcp4       0      0 *.13782            *.*              =
   LISTEN
 fffffe0132697b70 tcp4       0      0 127.0.0.1.51629    127.0.0.1.51632  =
   ESTABLISHED
 fffffe01326dbb70 tcp4       0      0 127.0.0.1.51632    127.0.0.1.51629  =
   ESTABLISHED
 fffffe0006f7d3d0 tcp4       0      0 *.13724            *.*              =
   LISTEN
 fffffe01326843d0 tcp4       0      0 *.22               *.*              =
   LISTEN
 fffffe01326847a0 tcp6       0      0 *.22               *.*              =
   LISTEN
 fffffe0006f7d000 tcp4       0      0 127.0.0.1.51629    *.*              =
   LISTEN
 fffffe0132153000 tcp4       0      0 127.0.0.1.1557     *.*              =
   LISTEN
 fffffe0006d95b70 tcp4       0      0 *.1556             *.*              =
   LISTEN
 fffffe01321533d0 tcp6       0      0 *.1556             *.*              =
   LISTEN
 fffffe0006342dc8 udp4       0      0 *.15949            *.*              =
  =20
 fffffe0006dcf930 udp6       0      0 *.*                *.*              =
  =20
 fffffe0006da2dc8 udp4       0      0 *.25350            *.*              =
  =20
 fffffe006b328498 udp6       0      0 *.*                *.*              =
  =20
 fffffe0006da2c40 udp4       0      0 195.234.155.143.58 8.8.8.8.53       =
  =20
 fffffe0006370000 udp4     631      0 195.234.155.143.95 8.8.8.8.53       =
  =20
 fffffe0006da2ab8 udp4       0      0 127.0.0.1.53       *.*              =
  =20
 fffffe0006375ab8 udp4       0      0 195.234.155.102.53 *.*              =
  =20
 fffffe00063497a8 udp4     149      0 *.56674            *.*              =
  =20
 fffffe0006370dc8 udp4     115      0 *.64883            *.*              =
  =20
 fffffe0006342498 udp4     128      0 *.43982            *.*              =
  =20
 fffffe0006dcfc40 udp4       0      0 127.0.0.1.123      *.*              =
  =20
 fffffe0006dcfdc8 udp6       0      0 fe80:5::1.123      *.*              =
  =20
 fffffe0006da2188 udp6       0      0 ::1.123            *.*              =
  =20
 fffffe0006da2498 udp6       0      0 fe80:4::5054:ff:.1 *.*              =
  =20
 fffffe0006371620 udp4       0      0 10.0.0.3.123       *.*              =
  =20
 fffffe0006371ab8 udp4       0      0 195.234.155.75.123 *.*              =
  =20
 fffffe0006375310 udp4       0      0 195.234.155.124.12 *.*              =
  =20
 fffffe0006375188 udp4       0      0 195.234.155.50.123 *.*              =
  =20
 fffffe0006375000 udp4       0      0 195.234.155.102.12 *.*              =
  =20
 fffffe0006371dc8 udp4       0      0 195.234.155.70.123 *.*              =
  =20
 fffffe0006371c40 udp6       0      0 fe80:3::5054:ff:.1 *.*              =
  =20
 fffffe0006371930 udp4       0      0 195.234.155.143.12 *.*              =
  =20
 fffffe00063717a8 udp6       0      0 fe80:2::216:8eff.1 *.*              =
  =20
 fffffe0006371188 udp4       0      0 192.168.122.3.123  *.*              =
  =20
 fffffe0006371310 udp6       0      0 *.123              *.*              =
  =20
 fffffe0006371498 udp4       0      0 *.123              *.*              =
  =20
 fffffe0006375c40 udp4       0      0 *.514              *.*              =
  =20
 fffffe0006375dc8 udp6       0      0 *.514              *.*              =
  =20
 Active UNIX domain sockets
 Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
 fffffe010a3f0690 stream      0      0        0 fffffe006be66e10        0 =
        0
 fffffe006be66e10 stream      0      0        0 fffffe010a3f0690        0 =
        0
 fffffe006be66c30 stream      0      0        0        0        0        =
 0
 fffffe010ab1dd20 stream      0      0        0 fffffe010ab1de10        0 =
        0
 fffffe010ab1de10 stream      0      0        0 fffffe010ab1dd20        0 =
        0
 fffffe010ab1e0f0 stream      0      0        0        0        0        =
 0
 fffffe006be67000 stream      0      0        0 fffffe006be674b0        0 =
        0
 fffffe006be674b0 stream      0      0        0 fffffe006be67000        0 =
        0
 fffffe006be67e10 stream      0      0        0        0        0        =
 0
 fffffe010ab69000 stream      0      0        0 fffffe010ab690f0        0 =
        0
 fffffe010ab690f0 stream      0      0        0 fffffe010ab69000        0 =
        0
 fffffe000662ac30 stream      0      0        0        0        0        =
 0
 fffffe000662a690 stream      0      0        0        0        0        =
 0
 fffffe006be665a0 stream      0      0        0        0        0        =
 0
 fffffe006be66780 stream      0      0        0 fffffe006be67780        0 =
        0
 fffffe006be67780 stream      0      0        0 fffffe006be66780        0 =
        0
 fffffe006b45d960 stream      0      0        0 fffffe00065703c0        0 =
        0
 fffffe00065703c0 stream      0      0        0 fffffe006b45d960        0 =
        0
 fffffe006be67870 stream      0      0        0 fffffe000656f4b0        0 =
        0
 fffffe000656f4b0 stream      0      0        0 fffffe006be67870        0 =
        0
 fffffe0006570960 stream      0      0        0        0        0        =
 0
 fffffe006be67d20 stream      0      0        0        0        0        =
 0
 fffffe01321d13c0 stream      0      0        0        0        0        =
 0
 fffffe006be66960 stream      0      0        0 fffffe006be661e0        0 =
        0
 fffffe006be661e0 stream      0      0        0 fffffe006be66960        0 =
        0
 fffffe01321d1a50 stream      0      0        0 fffffe006be67960        0 =
        0
 fffffe006be67960 stream      0      0        0 fffffe01321d1a50        0 =
        0
 fffffe006be662d0 stream      0      0        0 fffffe000656fd20        0 =
        0
 fffffe000656fd20 stream      0      0        0 fffffe006be662d0        0 =
        0
 fffffe01321d12d0 stream      0      0        0 fffffe01321d1960        0 =
        0
 fffffe01321d1960 stream      0      0        0 fffffe01321d12d0        0 =
        0
 fffffe010a3f04b0 stream      0      0        0        0        0        =
 0
 fffffe00065705a0 stream      0      0        0 fffffe01321d1d20        0 =
        0
 fffffe01321d1d20 stream      0      0        0 fffffe00065705a0        0 =
        0
 fffffe00066e3870 stream      0      0        0        0        0        =
 0
 fffffe006be66690 stream      0      0        0        0        0        =
 0
 fffffe010a3f00f0 stream      0      0        0 fffffe010a3f01e0        0 =
        0
 fffffe010a3f01e0 stream      0      0        0 fffffe010a3f00f0        0 =
        0
 fffffe006be66d20 stream      0      0        0        0        0        =
 0
 fffffe010a3f02d0 stream      0      0        0 fffffe010a3f03c0        0 =
        0
 fffffe010a3f03c0 stream      0      0        0 fffffe010a3f02d0        0 =
        0
 fffffe010a3f05a0 stream      0      0        0        0        0        =
 0
 fffffe00066e34b0 stream      0      0        0 fffffe006b45d3c0        0 =
        0
 fffffe006b45d3c0 stream      0      0        0 fffffe00066e34b0        0 =
        0
 fffffe00066e3000 stream      0      0        0 fffffe000662a5a0        0 =
        0
 fffffe000662a5a0 stream      0      0        0 fffffe00066e3000        0 =
        0
 fffffe010a3f0780 stream      0      0        0        0        0        =
 0
 fffffe006b45dc30 stream      0      0        0        0        0        =
 0
 fffffe006b45dd20 stream      0      0        0 fffffe006b45de10        0 =
        0
 fffffe006b45de10 stream      0      0        0 fffffe006b45dd20        0 =
        0
 fffffe010a3f0870 stream      0      0        0 fffffe010a3f0960        0 =
        0
 fffffe010a3f0960 stream      0      0        0 fffffe010a3f0870        0 =
        0
 fffffe0132a5dc30 stream      0      0        0        0        0        =
 0
 fffffe010a3f0a50 stream      0      0        0 fffffe010a3f0b40        0 =
        0
 fffffe010a3f0b40 stream      0      0        0 fffffe010a3f0a50        0 =
        0
 fffffe00066e33c0 stream      0      0        0        0        0        =
 0
 fffffe00065700f0 stream      0      0        0 fffffe006b45d4b0        0 =
        0
 fffffe006b45d4b0 stream      0      0        0 fffffe00065700f0        0 =
        0
 fffffe006b45d870 stream      0      0        0        0        0        =
 0
 fffffe006be663c0 stream      0      0        0 fffffe006be664b0        0 =
        0
 fffffe006be664b0 stream      0      0        0 fffffe006be663c0        0 =
        0
 fffffe00064cf4b0 stream      0      0        0        0        0        =
 0
 fffffe000662a4b0 stream      0      0        0 fffffe000662ab40        0 =
        0
 fffffe000662ab40 stream      0      0        0 fffffe000662a4b0        0 =
        0
 fffffe006be66a50 stream      0      0        0        0        0        =
 0
 fffffe000656fa50 stream      0      0        0 fffffe00066e30f0        0 =
        0
 fffffe00066e30f0 stream      0      0        0 fffffe000656fa50        0 =
        0
 fffffe00064cfd20 stream      0      0        0        0        0        =
 0
 fffffe006b45d5a0 stream      0      0        0 fffffe0132a5d1e0        0 =
        0
 fffffe0132a5d1e0 stream      0      0        0 fffffe006b45d5a0        0 =
        0
 fffffe0006570e10 stream      0      0        0        0        0        =
 0
 fffffe006b45da50 stream      0      0        0        0        0        =
 0
 fffffe00064cf2d0 stream      0      0        0 fffffe00064cf1e0        0 =
        0
 fffffe00064cf1e0 stream      0      0        0 fffffe00064cf2d0        0 =
        0
 fffffe006b45d690 stream      0      0        0        0        0        =
 0
 fffffe006be671e0 stream      0      0        0 fffffe006be672d0        0 =
        0
 fffffe006be672d0 stream      0      0        0 fffffe006be671e0        0 =
        0
 fffffe0006570b40 stream      0      0        0 fffffe00066e35a0        0 =
        0
 fffffe00066e35a0 stream      0      0        0 fffffe0006570b40        0 =
        0
 fffffe000662a3c0 stream      0      0        0        0        0        =
 0
 fffffe0132a5db40 stream      0      0        0 fffffe00066e3690        0 =
        0
 fffffe00066e3690 stream      0      0        0 fffffe0132a5db40        0 =
        0
 fffffe0006570c30 stream      0      0        0        0        0        =
 0
 fffffe0132a5d000 stream      0      0        0 fffffe01321d1e10        0 =
        0
 fffffe01321d1e10 stream      0      0        0 fffffe0132a5d000        0 =
        0
 fffffe0006570d20 stream      0      0        0        0        0        =
 0
 fffffe00066e31e0 stream      0      0        0 fffffe0006570a50        0 =
        0
 fffffe0006570a50 stream      0      0        0 fffffe00066e31e0        0 =
        0
 fffffe00064cf5a0 stream      0      0        0        0        0        =
 0
 fffffe000662a2d0 stream      0      0        0 fffffe01321d10f0        0 =
        0
 fffffe01321d10f0 stream      0      0        0 fffffe000662a2d0        0 =
        0
 fffffe000656f2d0 stream      0      0        0 fffffe000656f1e0        0 =
        0
 fffffe000656f1e0 stream      0      0        0 fffffe000656f2d0        0 =
        0
 fffffe00064cf0f0 stream      0      0        0 fffffe00064cf000        0 =
        0
 fffffe00064cf000 stream      0      0        0 fffffe00064cf0f0        0 =
        0
 fffffe000662ae10 stream      0      0        0 fffffe000662ad20        0 =
        0
 fffffe000662ad20 stream      0      0        0 fffffe000662ae10        0 =
        0
 fffffe0132a5e0f0 stream      0      0        0 fffffe0132a5e000        0 =
        0
 fffffe0132a5e000 stream      0      0        0 fffffe0132a5e0f0        0 =
        0
 fffffe000656f780 stream      0      0        0 fffffe000662a1e0        0 =
        0
 fffffe000662a1e0 stream      0      0        0 fffffe000656f780        0 =
        0
 fffffe00065704b0 stream      0      0        0 fffffe0132a5d5a0        0 =
        0
 fffffe0132a5d5a0 stream      0      0        0 fffffe00065704b0        0 =
        0
 fffffe000656f0f0 stream      0      0        0 fffffe00064cfa50        0 =
        0
 fffffe00064cfa50 stream      0      0        0 fffffe000656f0f0        0 =
        0
 fffffe00064cfe10 stream      0      0        0 fffffe00064cfb40        0 =
        0
 fffffe00064cfb40 stream      0      0        0 fffffe00064cfe10        0 =
        0
 fffffe0132a5d4b0 stream      0      0        0 fffffe0132a5d3c0        0 =
        0
 fffffe0132a5d3c0 stream      0      0        0 fffffe0132a5d4b0        0 =
        0
 fffffe0132a5d2d0 stream      0      0        0 fffffe000656f000        0 =
        0
 fffffe000656f000 stream      0      0        0 fffffe0132a5d2d0        0 =
        0
 fffffe000662aa50 stream      0      0        0 fffffe000662a960        0 =
        0
 fffffe000662a960 stream      0      0        0 fffffe000662aa50        0 =
        0
 fffffe000662a870 stream      0      0        0 fffffe000662a780        0 =
        0
 fffffe000662a780 stream      0      0        0 fffffe000662a870        0 =
        0
 fffffe0132a5e2d0 stream      0      0        0 fffffe0132a5e3c0        0 =
        0
 fffffe0132a5e3c0 stream      0      0        0 fffffe0132a5e2d0        0 =
        0
 fffffe00065702d0 stream      0      0        0 fffffe01321d14b0        0 =
        0
 fffffe01321d14b0 stream      0      0        0 fffffe00065702d0        0 =
        0
 fffffe0132a5d960 stream      0      0        0 fffffe00066e3a50        0 =
        0
 fffffe00066e3a50 stream      0      0        0 fffffe0132a5d960        0 =
        0
 fffffe0132a5e4b0 stream      0      0        0 fffffe0132a5e5a0        0 =
        0
 fffffe0132a5e5a0 stream      0      0        0 fffffe0132a5e4b0        0 =
        0
 fffffe00066e3b40 stream      0      0        0 fffffe000662a0f0        0 =
        0
 fffffe000662a0f0 stream      0      0        0 fffffe00066e3b40        0 =
        0
 fffffe0132a5e690 stream      0      0        0 fffffe0132a5e780        0 =
        0
 fffffe0132a5e780 stream      0      0        0 fffffe0132a5e690        0 =
        0
 fffffe000656fc30 stream      0      0        0 fffffe0132a5da50        0 =
        0
 fffffe0132a5da50 stream      0      0        0 fffffe000656fc30        0 =
        0
 fffffe00064cf3c0 stream      0      0        0 fffffe0132a5de10        0 =
        0
 fffffe0132a5de10 stream      0      0        0 fffffe00064cf3c0        0 =
        0
 fffffe0132a5ee10 stream      0      0        0 fffffe006b45d1e0        0 =
        0
 fffffe006b45d1e0 stream      0      0        0 fffffe0132a5ee10        0 =
        0
 fffffe006b45d0f0 stream      0      0        0 fffffe01321d11e0        0 =
        0
 fffffe01321d11e0 stream      0      0        0 fffffe006b45d0f0        0 =
        0
 fffffe0132a5e870 stream      0      0        0 fffffe0132a5e960        0 =
        0
 fffffe0132a5e960 stream      0      0        0 fffffe0132a5e870        0 =
        0
 fffffe0132a5ea50 stream      0      0        0 fffffe0132a5eb40        0 =
        0
 fffffe0132a5eb40 stream      0      0        0 fffffe0132a5ea50        0 =
        0
 fffffe0132a5ec30 stream      0      0 fffffe006b4e83c0        0        0 =
        0 /tmp/mysql.sock
 fffffe000656fb40 stream      0      0        0        0        0        =
 0
 fffffe0132a5e1e0 stream      0      0        0 fffffe0132a5d780        0 =
        0
 fffffe0132a5d780 stream      0      0        0 fffffe0132a5e1e0        0 =
        0
 fffffe01321d1b40 stream      0      0 fffffe006b4e5d20        0        0 =
        0 /var/run/authdaemond/socket.tmp
 fffffe0132a5d690 stream      0      0 fffffe006b1fc000        0        0 =
        0 /var/run/clamav/clamd
 fffffe000662a000 stream      0      0        0 fffffe01321d1870        0 =
        0
 fffffe01321d1870 stream      0      0        0 fffffe000662a000        0 =
        0
 fffffe00066e3c30 stream      0      0        0 fffffe00066e3d20        0 =
        0 /tmp/PBXPIPEbpcd
 fffffe00066e3d20 stream      0      0        0 fffffe00066e3c30        0 =
        0
 fffffe000656f690 stream      0      0 fffffe0132732780        0        0 =
        0 /usr/openv/var/vnetd/bpcd.uds
 fffffe01321d1690 stream      0      0 fffffe01326b53c0        0        0 =
        0 /usr/openv/var/vnetd/terminate_bpcd.uds
 fffffe00064cf960 stream      0      0        0 fffffe00065701e0        0 =
        0 /tmp/PBXPIPEvnetd
 fffffe00065701e0 stream      0      0        0 fffffe00064cf960        0 =
        0
 fffffe0006570000 stream      0      0 fffffe0132652000        0        0 =
        0 /usr/openv/var/vnetd/terminate_vnetd.uds
 fffffe00066e3e10 stream      0      0        0 fffffe01321d1000        0 =
        0
 fffffe01321d1000 stream      0      0        0 fffffe00066e3e10        0 =
        0
 fffffe000656fe10 stream      0      0 fffffe0006381960        0        0 =
        0 /var/run/devd.pipe
 fffffe00064cf780 dgram       0      0        0 fffffe000656f870        0 =
 fffffe00066e3780
 fffffe00066e3780 dgram       0      0        0 fffffe000656f870        0 =
 fffffe0132a5dd20
 fffffe0132a5dd20 dgram       0      0        0 fffffe000656f870        0 =
 fffffe00064cf870
 fffffe00064cf870 dgram       0      0        0 fffffe000656f870        0 =
 fffffe000656f5a0
 fffffe000656f5a0 dgram       0      0        0 fffffe000656f870        0 =
        0
 fffffe000656f870 dgram       0      0 fffffe00067035a0        0 =
 fffffe00064cf780        0 /var/run/logpriv
 fffffe000656f960 dgram       0      0 fffffe0006703780        0        0 =
        0 /var/run/log
 
 ------------------------------------------------------------------------
 netstat -aL
 
 Current listen queue sizes (qlen/incqlen/maxqlen)
 Proto Listen         Local Address        =20
 tcp4  0/0/20         localhost.26          =20
 tcp4  0/0/20         *.smtp                =20
 tcp4  0/0/20         localhost.domain      =20
 tcp4  0/0/50         *.submission          =20
 tcp4  0/0/50         localhost.mysql       =20
 tcp4  0/0/128        dentredjeb.4949       =20
 tcp46 0/0/128        *.pop3                =20
 tcp46 0/0/128        *.imaps               =20
 tcp46 0/0/128        *.imap                =20
 tcp4  0/0/128        *.https               =20
 tcp6  0/0/128        *.https               =20
 tcp4  0/0/128        *.xfer                =20
 tcp6  0/0/128        *.xfer                =20
 tcp4  0/0/128        *.http                =20
 tcp6  0/0/128        *.http                =20
 tcp4  0/0/64         *.ftp                 =20
 tcp4  0/0/5          *.13782               =20
 tcp4  0/0/5          *.13724               =20
 tcp4  0/0/128        *.ssh                 =20
 tcp6  0/0/128        *.ssh                 =20
 tcp4  0/0/5          localhost.51629       =20
 tcp4  0/0/5          localhost.1557        =20
 tcp4  0/0/5          *.1556                =20
 tcp6  0/0/5          *.1556                =20
 unix  0/0/50         /tmp/mysql.sock
 unix  0/0/128        /var/run/authdaemond/socket.tmp
 unix  0/0/128        /var/run/clamav/clamd
 unix  0/0/5          /usr/openv/var/vnetd/bpcd.uds
 unix  0/0/1          /usr/openv/var/vnetd/terminate_bpcd.uds
 unix  0/0/1          /usr/openv/var/vnetd/terminate_vnetd.uds
 unix  0/0/4          /var/run/devd.pipe
 
 ------------------------------------------------------------------------
 fstat
 [root@dentredje crash]#
 ***
 
 
 On 05/03/2013, at 17.56, Rasmus Skaarup <freebsd@gal.dk> wrote:
 
 >=20
 > kgdb) frame 4
 > #4  0xffffffff816bd5ba in zfs_freebsd_read (ap=3DVariable "ap" is not =
 available.
 > )
 >    at =
 /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/=
 zfs_vnops.c:476
 > warning: Source file is more recent than executable.
 >=20
 > 476					if (pp->wire_count =3D=3D 0 && =
 pp->valid =3D=3D 0 &&
 > (kgdb) info reg
 > rax            0x0	0
 > rbx            0x7a	122
 > rcx            0x0	0
 > rdx            0x0	0
 > rsi            0x0	0
 > rdi            0x0	0
 > rbp            0xffffff8162323810	0xffffff8162323810
 > rsp            0xffffff8162323710	0xffffff8162323710
 > r8             0x0	0
 > r9             0x0	0
 > r10            0x0	0
 > r11            0x0	0
 > r12            0xfffffe015123cc80	-2193366987648
 > r13            0xfffffe0006aa1000	-2198911447040
 > r14            0xfffffe0006aa10e8	-2198911446808
 > r15            0xffffff8162323850	-543813388208
 > rip            0xffffffff816bd5ba	0xffffffff816bd5ba =
 <zfs_freebsd_read+1578>
 > eflags         0x0	0
 > cs             0x0	0
 > ss             0x0	0
 > ds             0x0	0
 > es             0x0	0
 > fs             0x0	0
 > gs             0x0	0
 > (kgdb)=20
 >=20
 >=20
 > I guess line 476 changed after your patch, but I guess you can use the =
 output anyway.
 >=20
 > Br
 > Rasmus Skaarup
 >=20
 > On 05/03/2013, at 17.51, Andriy Gapon <avg@FreeBSD.org> wrote:
 >=20
 >>=20
 >> I need 'info reg' from zfs_freebsd_read frame, which frame 4 in kgdb.
 >>=20
 >>=20
 >> --=20
 >> Andriy Gapon
 >>=20
 >=20
 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201303061250.r26Co1jN079370>