Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 2 Feb 2014 20:46:23 +0100
From:      Matthew Rezny <matthew@reztek.cz>
To:        freebsd-stable@freebsd.org
Subject:   Tuning kern.maxswzone is minor compared to hangs in "kmem a" state
Message-ID:  <20140202204623.00003fe5@unknown>
In-Reply-To: <20140201221612.00001897@unknown>
References:  <20140201070912.00007971@unknown> <x7d2j64pvw.fsf@ichotolot.servalan.com> <20140201221612.00001897@unknown>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 1 Feb 2014 22:16:12 +0100
Matthew Rezny <matthew@reztek.cz> wrote:

> On Sat, 01 Feb 2014 13:03:15 -0600
> Richard Todd <rmtodd@servalan.servalan.com> wrote:
> 
> > Matthew Rezny <matthew@reztek.cz> writes:
> > 
> > > So, as best I can tell, the actual effective number used for
> > > kern.maxswzone is indeed approximately 8x available RAM. If there
> > > is some need to turn it down (using substantially less swap) then
> > > that is possible, but turning it up (as suggested by the warning
> > > message) is not possible. Setting any value higher does not appear
> > > to actually
> > 
> > Yeah, IIRC I ran into that when configing some VMs with
> > really big swap space for benefit of tmpfs.   This is the quick hack
> > I used to get around that, you might give it a try. 
> > 
> > # HG changeset patch
> > # Parent e4dd7df011139e2b224835aa6e330c90afcf9a55
> > patch swap_pager to unconditionally use maxswzone tunable if it is
> > set -- helpful for our tbvm VMs with large swap space for tmpfs
> > 
> > diff -r e4dd7df01113 sys/vm/swap_pager.c
> > --- a/sys/vm/swap_pager.c	Wed Feb 20 00:15:49 2013 -0600
> > +++ b/sys/vm/swap_pager.c	Sat Feb 23 13:08:54 2013 -0600
> > @@ -546,7 +546,7 @@
> >  	 * is typically limited to around 32MB by default.
> >  	 */
> >  	n = cnt.v_page_count / 2;
> > -	if (maxswzone && n > maxswzone / sizeof(struct swblock))
> > +	if (maxswzone)
> >  		n = maxswzone / sizeof(struct swblock);
> >  	n2 = n;
> >  	swap_zone = uma_zcreate("SWAPMETA", sizeof(struct swblock),
> > NULL, NULL,
> > 
> Thank you for pointing me in the right direction. Now that I'm
> looking at the right file, I don't know how I failed to find it
> myself with grep. The logic here is rather obviously flawed, the
> maxswzone value is only used if it is less than the calculated
> default. If there is some reason to not allow adjusting this up, then
> the warning message is incorrect. Either way, something should change
> and I guess I should file a PR on this one. At least I can see the
> warning is taking the doubling for safety into account, so for my
> particular case with an overrun under 10% it should probably be ok to
> put fixing this on the back burner.
> 
> > Also,
> > 
> > > With /usr/src cleared (and after running fsck) I booted back into
> > > 10-PRERELEASE to try to fetch the 10-STABLE sources again. I
> > > started svnlite co and find it hung shortly thereafter. I tried a
> > > few times but each time I see it does a couple hundred files at
> > > best and just stops. When it stops, I can't login to another
> > > terminal. If I have a spare console logged in, I can't run
> > > anything. After a few tries, I manged to catch it where I had top
> > > running in one VT, started the checkout, and then switched back
> > > just in time. I never could even get top up with rm running (it
> > > probably blows over some limit faster). When the checkout hangs,
> > > the state of svnlite is "kmem a" according to top. I can only
> > > guess that is short for kmem alloc, I guess some syscall is
> > > waiting on an allocation that will never happen because something
> > > already is using or has leaked everything that could satisfy that
> > > allocation. It looks like activity on too many files within a
> > > short period runs something out.
> > 
> > No, it's just a new bit of debugging code that causes the system
> > to spend lots of CPU time verifying integrity of some of its
> > internal data structures, especially on wimpy hardware (e.g., my
> > dual PII/400 box, which is where I noticed this recently.)  You'll
> > find if you're patient that it isn't a complete hang, it will
> > actually get work done in between the debug passes. Set sysctl
> > debug.vmem_check=0 to disable the check.  This is I think
> > completely independent from the maxswzone stuff, it's just you were
> > seeing it for the first time since the debug code in question was
> > only recently added to 10-STABLE.
> > 
> If only it were that simple. I'm not yet on 10-STABLE, I'm struggling
> to get the sources to reach that point. These C3 boxes are
> 10-PRERELEASE so I don't yet have this debug.vmem_check to tweak.
> sysctl says unknown oid when I query it. I tried setting it from
> loader prompt but still says unknown oid and I see no change in
> behavior.
> 
> Also, I'm not seeing anything using lots of CPU time. If I start off
> top on another VT before I start svnlite, then I have a decent chance
> of seeing what goes on until the situation becomes dire. svnlite
> starts off moving quick and using lots of CPU time (>50%) for the
> first hundred files or so, then it halts in the "kmem a" state and
> CPU is completely idle. It sits there for a while, and eventually does
> something more. Each time the stop is longer and less work is done in
> the interval between stops. Eventually the process appears to hang
> completely in that I can leave it for half a day and no more progress
> is made. The rm process could go longer in this state with still some
> visible progress since it's operation is sufficiently simple to
> actually observe it managed to do something, whereas svnlite might do
> something occasionally but it's not enough to get to the next file in
> the list.
> 
> With each burst and wait cycle, I see a spray in dmesg saying
> calcru: runtime went backwards [lots] for {all processes}. If I hit
> ctrl-C and wait, it'll interrupt svnlite the next time it would do
> something other than wait. It can easily take 10 min for that to
> happen, meaning the wait period is long enough to consider the process
> as not effectively running. If I let it go long enough, it gets to a
> state where it never exits on ctrl-c (or at least it'd take more hours
> than I've been willing to wait). In this last attempt, I let it go for
> 5 min, pushed ctrl-c, waited another 10 min, then tried ctrl-alt-del
> and that just beeps so the attempt to interrupt the process resulted
> in total I/O lockup to the point key presses are not handled. That
> last one was enough to finally require manual fsck on reboot (which
> should be a testament to the resiliency of UFS as I've pushed the
> reset button a hundred times in the past day and a half).
> 
> >                         Richard
> > 
> > 
> 
> Thank you for taking the time to respond. It's now clear the maxswzone
> is just a red herring, the real issue are the apparent hangs which I'm
> seeing on several more boxes. The mystery is why the one box with
> slightly more RAM seems ok, but a couple boxes with far more RAM are
> not ok. That will probably be answered when I figure out what the
> cause is.

Reply to self for status update.

The situation with these C3 boxes and 10.0 is just miserable. I tried
using tar, both streaming with netcat and from a file, to populate
/usr/src on the smaller box, but tar would hang eventually too.
Tar got further probably because it's only handling the source files
and not constantly updating a database file as svn does. I decided to
try running svnlite through truss to see if there was a particular
syscall causing trouble. In two runs I saw that it got further than
ever before (likely truss output was limiting overall execution speed
and thus reducing peak instantaneous disk I/O load) but stopped at
different points. The first attempt hung on a stat() call on wc.db and I
saw that a call involving the same file just a few lines before had
failed with file not found, but that file must exist and indeed was
present (but corrupt) following a reboot and fsck. The second run
through stopped on a blank line as if the next syscall was setup but
the name didn't get printed before everything hung hard. Following the
pattern of 5 gettimeofday() between each lstat(), the next call should
have been an lstat() but on which file is unknown because each lstat()
prior was a different file. It looks like the hangs might be connected
to errors opening files that do exist. That bit of info ties this in
another way to the P4 boxes mentioned in my other thread, which simply
failed out of builworld with claims source files didn't exist, but
completed buildworld on the next attempt.

After deciding there was just no way to have significant disk I/O on
the box with only 256MB RAM, I moved it's harddrive to the box with
384MB and extracted the tar there, That succeeded on the first attempt,
which verifies the UFS volume hasn't been horribly trashed in some way
that manifests as file access related hangs. Booting the drive in the
other system also verifies it is not any minor accidental configuration
difference as now all that changed was the available memory and
suddenly the same OS instance could extract the tar of both /usr/src
and /usr/ports no problem. I put the harddrives back in their
respective boxes and booted them both up. On each I started off top on
one VT and then went to another VT and ran svnlite status to verify
that svn regards the source tree as complete. On the box with 384MB
RAM, that command returned the expected result in under a minute. On
the box with 256MB RAM, I saw a quick burst of disk I/O and then
nothing. Top confirmed svnlite was sitting in "kmem a" state. I hit
ctrl-c and let it sit. An hour later it was sitting in the same place
(so zero progress) and I went to sleep. When I awoke some 6 hours
later, it was in the same state with the same 9sec CPU time, the
process definitely isn't even creeping along with periodic progress
like rm sometimes manages. Looking at top revealed that ntpd has joined
svnlite in the "kmem a" state. I had noticed yesterday the clock was
off by several hours. Now I know why, if I leave it hung too long ntpd
gets hung and the clock grinds to a halt. How could a process hanging
stop the RTC on the mobo? Only thing I can think of is that might
happen if ntpd were to hang in the midst of an update to the RTC. I
wonder if all the calcru: runtime went backwards errors are related to
periodic clock stops.

I rebooted the hung box and fsck showed nothing significant. I started
top on both (to ensure it's available) and then started a make
buildworld on each. On the box with 256MB, that hung in the initial rm
-rf /usr/obj/usr/src/tmp. Yep, that should be no surprise, the old
/usr/obj is still there and it's too big to rm. Using mfsbsd I was
able to clear that directory with just a few tries (same as before, rm
causes pressure fast enough to blow out sh, getty and everything short
of init). Booted back into the installed OS, I started the buildworld
again and it had been making reasonable progress for a few minutes.
During compilation, file I/O will be bursts with long waits between as
the CPU churns. My amd64 boxes are fast enough to be bottlenecked on
disk I/O unless I build to a tmpfs. These C3 boxes are so slow the
compiler could never hope to saturate the disk, but a big copy or an
mtree might blow it up the same as the initial rm.

Just as I was about to hit send, the box panicked. Finally a panic and
not just a hang so there's a backtrace! That should be the first hint
as to where in the kernel I'm hitting trouble. The panic points the
finger at UFS. The filesystem has seen some thrashing but it did just
fsck without significant error before that build was started. So, I'm
thinking it's not on-disk corruption but an in-memory problem. It blew
up writing a softdep, which starts me wondering if the buffers used for
softupdates bookkeeping are overflowing in the kernel. That would make
some sense, rapidly creating (svn/tar) or deleting (rm) many small
files will create a huge amount of activity for softupdates to track,
defer and reorder. The rm case is probably the one than can trigger it
the fastest because deletes are focusing the beating on the metadata
without wasting time between on file data I/O.

Here's the full core.txt in hopes the vmstat numbers add some useful
information beyond the backtrace.

Sun Feb  2 20:07:30 CET 2014

FreeBSD music2.reztek 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #1: Sat Jan 11 09:00:08 CET 2014     root@music2.reztek:/usr/obj/usr/src/sys/MUSIC  i386

panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00
KDB: stack backtrace:
#0 0xc0667f6f at kdb_backtrace+0x52
#1 0xc063dc7d at panic+0x85
#2 0xc084483f at softdep_disk_write_complete+0x1294
#3 0xc06b0656 at bufdone_finish+0x23
#4 0xc06b04f5 at bufdone+0x39
#5 0xc05e0d13 at g_vfs_done+0x289
#6 0xc05dd8ea at g_io_deliver+0x28f
#7 0xc05e0648 at g_std_done+0x4e
#8 0xc05dd8ea at g_io_deliver+0x28f
#9 0xc05e0648 at g_std_done+0x4e
#10 0xc05dd8ea at g_io_deliver+0x28f
#11 0xc05e0648 at g_std_done+0x4e
#12 0xc05dd8ea at g_io_deliver+0x28f
#13 0xc05db68f at g_disk_done_single+0xfa
#14 0xc0487fb7 at adadone+0x482
#15 0xc046de75 at xpt_done_process+0x435
#16 0xc0470a11 at xpt_done_td+0x15a
#17 0xc0615573 at fork_exit+0xa3
Uptime: 6m58s
Physical memory: 239 MB
Dumping 102 MB: 87 71 55 39 23 7

No symbol "stopped_cpus" in current context.
#0  doadump (textdump=-1030183936) at pcpu.h:233
233	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=-1030183936) at pcpu.h:233
#1  0xc063d96b in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xc063dce6 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xc084483f in softdep_disk_write_complete (bp=0x0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:11270
#4  0xc06b0656 in bufdone_finish (bp=0xc1ecfbe0) at buf.h:420
#5  0xc06b04f5 in bufdone (bp=<value optimized out>)
    at /usr/src/sys/kern/vfs_bio.c:3754
#6  0xc05e0d13 in g_vfs_done (bip=<value optimized out>)
    at /usr/src/sys/geom/geom_vfs.c:157
#7  0xc05dd8ea in g_io_deliver (bp=0xc724f390, error=<value optimized out>)
    at /usr/src/sys/geom/geom_io.c:669
#8  0xc05e0648 in g_std_done (bp=<value optimized out>)
    at /usr/src/sys/geom/geom_subr.c:1018
#9  0xc05dd8ea in g_io_deliver (bp=0xc58f5000, error=<value optimized out>)
    at /usr/src/sys/geom/geom_io.c:669
#10 0xc05e0648 in g_std_done (bp=<value optimized out>)
    at /usr/src/sys/geom/geom_subr.c:1018
#11 0xc05dd8ea in g_io_deliver (bp=0xc724f980, error=<value optimized out>)
    at /usr/src/sys/geom/geom_io.c:669
#12 0xc05e0648 in g_std_done (bp=<value optimized out>)
    at /usr/src/sys/geom/geom_subr.c:1018
#13 0xc05dd8ea in g_io_deliver (bp=0xc724fb48, error=<value optimized out>)
    at /usr/src/sys/geom/geom_io.c:669
#14 0xc05db68f in g_disk_done_single (bp=0xc724fb48)
    at /usr/src/sys/geom/geom_disk.c:275
#15 0xc0487fb7 in adadone (periph=<value optimized out>, 
    done_ccb=<value optimized out>) at /usr/src/sys/cam/ata/ata_da.c:1759
#16 0xc046de75 in xpt_done_process (ccb_h=0xc564c800)
    at /usr/src/sys/cam/cam_xpt.c:5247
#17 0xc0470a11 in xpt_done_td (arg=0xc09c3e00)
    at /usr/src/sys/cam/cam_xpt.c:5278
#18 0xc0615573 in fork_exit (callout=0xc04708b7 <xpt_done_td>)
    at /usr/src/sys/kern/kern_fork.c:995
#19 0xc08bd154 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID  PID PPID CPU PRI NI   VSZ     RSS MWCHAN   STAT TT     TIME COMMAND
  0    0    0   0  -8  0     0       0 -        DLs   -  0:00.01 [kernel]
  0    1    0   0   8  0  9240 3154304 wait     DLs   -  0:00.03 [init]
  0    2    0   0   8  0     0       0 crypto_w DL    -  0:00.00 [crypto]
  0    3    0   0   8  0     0       0 crypto_r DL    -  0:00.00 [crypto returns]
  0    4    0   0  -8  0     0       0 ccb_scan DL    -  0:03.55 [cam]
  0    5    0   0  -8  0     0       0 ctl_work DL    -  0:00.00 [ctl_thrd]
  0    6    0   0 -16  0     0       0 waiting_ DL    -  0:00.00 [sctp_iterator]
  0    7    0   0 -16  0     0       0 psleep   DL    -  0:00.83 [pagedaemon]
  0    8    0   0  16  0     0       0 psleep   DL    -  0:00.08 [vmdaemon]
  0    9    0   0 155  0     0       0 pgzero   DL    -  0:00.00 [pagezero]
  0   10    0   0 -16  0     0       0 audit_wo DL    -  0:00.00 [audit]
  0   11    0   0 155  0     0       0 -        RL    -  1:48.78 [idle]
  0   12    0   0 -84  0     0       0 -        WL    -  0:11.94 [intr]
  0   13    0   0  -8  0     0       0 -        DL    -  0:00.05 [geom]
  0   14    0   0 -16  0     0       0 -        DL    -  0:00.65 [rand_harvestq]
  0   15    0   0 -68  0     0       0 -        DL    -  0:00.01 [usb]
  0   16    0   0 -16  0     0       0 psleep   DL    -  0:00.36 [bufdaemon]
  0   17    0   0  16  0     0       0 syncer   DL    -  0:00.18 [syncer]
  0   18    0   0  -4  0     0       0 vlruwt   DL    -  0:00.01 [vnlru]
  0   19    0   0 -16  0     0       0 sdflush  DL    -  0:00.30 [softdepflush]
  0   20    0   0 -16  0     0       0 -        DL    -  0:00.06 [schedcpu]
  0 1090    1   0  60  0 10048 3151424 select   Ds    -  0:00.01 [dhclient]
 65 1126    1   0  55  0 10048 3155264 select   Ds    -  0:00.00 [dhclient]
  0 1127    1   0  40  0  9224 3152384 select   Ds    -  0:00.00 [devd]
  0 1274    1   0  40  0  9944 3153344 select   Ds    -  0:00.39 [syslogd]
  0 1369    1   0  40  0 12032 3148544 select   Ds    -  0:00.15 [ntpd]
  0 1398    1   0  59  0 14312 3149504 select   Ds    -  0:00.00 [sshd]
  0 1402    1   0   8  0  9984 3146624 nanslp   Ds    -  0:00.12 [cron]
  0 1425    1   0  59  0 10068 3158144 select   Ds    -  0:00.00 [moused]
  0 1468    1   0  58  0  9936 3176448 ttyin    Ds+   -  0:00.01 [getty]
  0 1469    1   0   8  0 10416 3174528 wait     DWs   -  0:00.00 [login]
  0 1470    1   0   8  0 10416 3177408 wait     DWs   -  0:00.00 [login]
  0 1471    1   0  58  0  9936 3150464 ttyin    Ds+   -  0:00.01 [getty]
  0 1472    1   0  58  0  9936 3172608 ttyin    Ds+   -  0:00.01 [getty]
  0 1473    1   0  58  0  9936 3173568 ttyin    Ds+   -  0:00.01 [getty]
  0 1474    1   0  58  0  9936 3171648 ttyin    Ds+   -  0:00.01 [getty]
  0 1475    1   0  58  0  9936 3170688 ttyin    Ds+   -  0:00.01 [getty]
  0 1476 1470   0  16  0 10656 3169728 pause    DW    -  0:00.00 [csh]
  0 1478 1476   0  40  0 11056 3168768 select   D+    -  0:02.19 [top]
  0 1479 1469   0  16  0 10656 3167808 pause    DW    -  0:00.00 [csh]
  0 1483 1479   0   8  0  8944 3166848 wait     DW+   -  0:00.00 [make]
  0 1488 1483   0   8  0  8944 3165888 wait     DW+   -  0:00.00 [make]
  0 1619 1488   0   8  0  8944 3164928 wait     DW+   -  0:00.00 [make]
  0 1623 1619   0   8  0 10272 3163968 wait     DW+   -  0:00.00 (sh)
  0 1725 1623   0   8  0  8944 3147584 wait     D+    -  0:00.76 [make]
  0 1793 1725   0   8  0 27316 3156224 wait     D+    -  0:00.04 [c++]
  0 1794 1793   0 -16  0 37300 4128768 vnread   DL+   -  0:06.44 [c++]

------------------------------------------------------------------------
vmstat -s

   201483 cpu context switches
   867407 device interrupts
    21239 software interrupts
   229915 traps
   396065 system calls
       20 kernel threads created
     1419  fork() calls
      355 vfork() calls
        0 rfork() calls
        9 swap pager pageins
       33 swap pager pages paged in
       52 swap pager pageouts
      153 swap pager pages paged out
     1558 vnode pager pageins
    10442 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
      108 page daemon wakeups
        0 pages examined by the page daemon
      116 pages reactivated
    37654 copy-on-write faults
      248 copy-on-write optimized faults
   144312 zero fill pages zeroed
     3353 zero fill pages prezeroed
        9 intransit blocking page faults
   201812 total VM faults taken
     1537 page faults requiring I/O
        0 pages affected by kernel thread creation
    54059 pages affected by  fork()
    16118 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   238064 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
    10588 pages active
      265 pages inactive
        8 pages in VM cache
    20998 pages wired down
    28211 pages free
     4096 bytes per page
   504005 total name lookups
          cache hits (89% pos + 1% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       ctlmem  9179 18345K       -     9179  64,2048
         cdev    10     2K       -       10  128
      CAM XPT    25     2K       -       85  16,32,128,256,512,1024,2048
       ctlblk   200   800K       -      200  4096
     filedesc    49    52K       -     1797  16,1024,4096
        sigio     1     1K       -        1  32
     filecaps     1     1K       -        3  16
      kdtrace   125    23K       -     1873  64,256
         kenv    82     7K       -       93  16,32,64,128,4096
       kqueue     0     0K       -       12  128,1024
    proc-args    24     2K       -      610  16,32,64,128,256
      ramdisk     1  4096K       -        1  
        hhook     2     1K       -        2  128
      ithread    59     5K       -       59  16,64,128
        mixer     1     4K       -        1  4096
       KTRACE   100    13K       -      100  128
       kbdmux     6    18K       -        6  16,256,1024,2048
       linker   106     4K       -      108  16,32,256
        lockf    17     1K       -       49  32,64
   loginclass     3     1K       -       16  64
        cache     1     1K       -        1  16
       devbuf   810 45406K       -     1186  16,32,64,128,256,512,1024,2048,4096
         temp    18    77K       -     1696  16,32,64,128,256,512,1024,2048,4096
       ip6ndp     5     1K       -        5  64,128
       module   220    14K       -      220  64
     mtx_pool     2     8K       -        2  4096
     pmchooks     1     1K       -        1  64
         UART    12     7K       -       12  16,512,1024
      CAM DEV     6    12K       -       12  2048
         pgrp    21     2K       -       26  64
      session    17     3K       -       22  128
         proc     2     2K       -        2  1024
      subproc   114   205K       -     1862  256,4096
         cred    46     5K       -     6934  64,128
       plimit    12     3K       -      209  256
      uidinfo     3     1K       -        6  64,256
      CAM CCB     0     0K       -    32459  2048
     CAM path     8     1K       -       50  16
      ctlpool   532   138K       -      532  16,512
          USB     6     3K       -        6  16,64,256,2048
       USBdev     5     1K       -        5  32,64,128
       sysctl     0     0K       -     2612  16,32,64
    sysctloid  2164    66K       -     2215  16,32,64,128
    sysctltmp     0     0K       -      210  16,32,64,128
   CAM periph     6     1K       -       21  16,32,64,128
      tidhash     1     2K       -        1  2048
      callout     2   480K       -        2  
         umtx   180    17K       -      180  64,128
     p1003.1b     1     1K       -        1  16
         SWAP     2   277K       -        2  64
          bus   457    21K       -     1529  16,32,64,128,256,1024
       bus-sc    44    60K       -      564  16,32,64,128,256,512,1024,2048,4096
    CAM queue    10     2K       -       30  16,256
      devstat     6    13K       -        6  16,4096
 eventhandler    82     4K       -       82  32,64,128
       DEVFS3   134    17K       -      147  128
         kobj   117   234K       -      154  2048
       DEVFS1   111    28K       -      117  256
      Per-cpu     1     1K       -        1  16
        DEVFS    17     1K       -       18  16,64
       DEVFSP     2     1K       -        2  32
         rman   100     6K       -      523  16,32,64
         sbuf     0     0K       -      646  16,32,64,128,256,512,1024,2048,4096
    taskqueue    13     2K       -       13  16,128
       Unitno    22     2K       -      320  16,64
         vmem     4    38K       -       11  128,512,1024,2048,4096
     ioctlops     0     0K       -     1297  16,32,64,128,256,512,1024
       select    10     1K       -       10  64
          iov     0     0K       -    79818  16,64,128,256
          msg     4    25K       -        4  1024,4096
          sem     4   101K       -        4  1024,4096
          shm     1    12K       -        1  
          tty    21    11K       -       21  512
     mbuf_tag     0     0K       -       18  32
         ksem     1     4K       -        1  4096
        shmfd     1     4K       -        1  4096
       soname     3     1K       -     9384  16,32,128
          pcb    19    91K       -       54  16,64,512,1024,4096
     vfscache     1   128K       -        1  
   cl_savebuf     0     0K       -        7  32
     vfs_hash     1    64K       -        1  
       vnodes     1     1K       -        1  128
        mount    34     2K       -      143  16,32,64,128,256
  vnodemarker     0     0K       -       42  512
      GSS-API     1     1K       -        1  32
          BPF    12    18K       -       12  64,128,512,4096
        ifnet     5     5K       -        5  64,1024
       ifaddr    53    10K       -       54  16,32,64,128,256,512,1024,2048
  ether_multi    17     1K       -       24  16,32,64
        clone     8     1K       -        8  128
       arpcom     1     1K       -        1  16
      lltable    12     3K       -       12  256
     routetbl    33     4K       -      171  16,32,64,128,256
         igmp     4     1K       -        4  128
     in_multi     2     1K       -        3  128
encap_export_host     2     2K       -        2  1024
    sctp_a_it     0     0K       -        3  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     4     1K       -        4  128
     sctp_ifn     2     1K       -        2  128
    sctp_iter     0     0K       -        3  256
    hostcache     1    16K       -        1  
     syncache     1    44K       -        1  
    in6_multi    15     2K       -       15  16,256
          mld     4     1K       -        4  128
  inpcbpolicy    11     1K       -      179  16
  ipsecpolicy    22     3K       -      358  128
  IpFw/IpAcct    17    31K       -       28  16,32,64,128,512
      NFS FHA     1     1K       -        1  1024
       crypto     1     1K       -        1  512
          rpc     2     1K       -        2  128
audit_evclass   187     3K       -      228  16
      pagedep     2     9K       -     1649  128
     inodedep    25    70K       -     4100  256
    bmsafemap     5     5K       -     1837  128,4096
       newblk    24   131K       -     3278  128
     indirdep     0     0K       -      124  64
     freefrag     2     1K       -       48  64
     freeblks     4     1K       -      717  128
     freefile     6     1K       -     1687  32
       diradd    16     1K       -     1742  64
        mkdir     0     0K       -     3274  64
       dirrem     0     0K       -       79  64
    newdirblk     0     0K       -     1639  32
     freework    18     3K       -      834  128,256
        sbdep     0     0K       -       16  32
     savedino     8     2K       -     1223  256
      softdep     1     1K       -        1  256
  ufs_dirhash    29    10K       -       89  16,32,64,512
    ufs_mount     4    13K       -        8  64,256,1024,4096
    vm_pgdata     2    17K       -        2  64
  nullfs_hash     1    64K       -        1  
     atkbddev     2     1K       -        2  32
    pfs_nodes    21     3K       -       21  128
  pfs_vncache     1     1K       -        1  32
         GEOM   113    21K       -      505  16,32,64,128,256,512,1024
     ppbusdev     3     1K       -        3  128
      memdesc     1     4K       -        1  4096
      entropy  1026    65K       -     1028  16,64,4096
CAM dev queue     4     1K       -        4  64
         $PIR     6     1K       -        6  32
     envy24ht    15   194K       -       15  32,64,1024
       spicds     7     1K       -        7  64
      CAM SIM     4     1K       -        4  128
          agp     2    65K       -        2  16
       isadev    21     2K       -       21  64
    legacydrv     4     1K       -        4  16
     nexusdev     4     1K       -        4  16
       feeder    16     1K       -       18  16,64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               256,      0,     106,      14,     106,   0,   0
UMA Zones:              408,      0,     106,       2,     106,   0,   0
UMA Slabs:               56,      0,    5122,      62,    5573,   0,   0
UMA RCntSlabs:           64,      0,     319,      59,     374,   0,   0
UMA Hash:               128,      0,       3,      28,       3,   0,   0
4 Bucket:                16,      0,       6,     246,     381,   0,   0
6 Bucket:                24,      0,       0,       0,       0,   0,   0
8 Bucket:                32,      0,       2,     124,    2021,   0,   0
12 Bucket:               48,      0,       0,       0,       0,   0,   0
16 Bucket:               64,      0,       8,     118,      48,   0,   0
32 Bucket:              128,      0,       9,     146,    2054,   0,   0
64 Bucket:              256,      0,      14,      61,     215,   0,   0
128 Bucket:             512,      0,      62,       2,     795,3846,   0
vmem btag:               28,      0,    7567,     497,    8975, 110,   0
VM OBJECT:              156,      0,    1515,      60,   18171,   0,   0
RADIX NODE:              44,  59605,    5330,    4953,  122443,   0,   0
MAP:                    144,      0,       3,      81,       3,   0,   0
KMAP ENTRY:              76,      0,       5,     154,       5,   0,   0
MAP ENTRY:               76,      0,     478,     158,   40973,   0,   0
VMSPACE:                240,      0,      27,      69,    1776,   0,   0
fakepg:                  68,      0,       0,       0,       0,   0,   0
mt_zone:                 76,      0,     299,      72,     299,   0,   0
16:                      16,      0,    2152,     368,    8788,   0,   0
32:                      32,      0,     944,     442,    7929,   0,   0
64:                      64,      0,    2449,     638,   87186,   0,   0
128:                    128,      0,    1115,     466,   26670,   0,   0
256:                    256,      0,     396,     144,    6359,   0,   0
512:                    512,      0,     340,      36,    1128,   0,   0
1024:                  1024,      0,      84,      32,    2220,   0,   0
2048:                  2048,      0,    9321,       9,   41863,   0,   0
4096:                  4096,      0,     273,       4,    2055,   0,   0
uint64 pcpu:              8,      0,    2301,     219,    2301,   0,   0
SLEEPQUEUE:              44,      0,      91,     161,      91,   0,   0
Files:                   56,      0,      61,     155,   53221,   0,   0
rl_entry:                28,      0,      34,     254,      34,   0,   0
TURNSTILE:               72,      0,      91,     126,      91,   0,   0
umtx pi:                 52,      0,       0,       0,       0,   0,   0
MAC labels:              20,      0,       0,       0,       0,   0,   0
PROC:                   748,      0,      46,      19,    1794,   0,   0
THREAD:                 760,      0,      77,      13,      77,   0,   0
cpuset:                  40,      0,      45,     157,      45,   0,   0
audit_record:          1112,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,  32085,     512,     137,     567,   0,   0
mbuf:                   256,  32085,       1,     262,    6795,   0,   0
mbuf_cluster:          2048,   5014,     637,       1,     750,   0,   0
mbuf_jumbo_page:       4096,   2506,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216,   2226,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   1668,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
ttyoutq:                256,      0,      64,      41,     256,   0,   0
g_bio:                  152,      0,      27,     103,  129555,   0,   0
icl_conn:                88,      0,       0,       0,       0,   0,   0
icl_pdu:                 48,      0,       0,       0,       0,   0,   0
ttyinq:                 152,      0,     120,      10,     480,   0,   0
ata_request:            220,      0,       1,      17,   34069,   0,   0
cryptop:                 60,      0,       0,       0,       0,   0,   0
cryptodesc:              56,      0,       0,       0,       0,   0,   0
cfiscsi_data_wait:       32,      0,       0,       0,       0,   0,   0
VNODE:                  284,      0,    2752,      20,    4442,   0,   0
VNODEPOLL:               60,      0,       0,       0,       0,   0,   0
BUF TRIE:                44,      0,      33,    2242,    8894,   0,   0
NAMEI:                 1024,      0,       0,      16,   87795,   0,   0
S VFS Cache:             72,      0,    2886,     138,   45560,   0,   0
STS VFS Cache:           92,      0,       0,       0,       0,   0,   0
L VFS Cache:            292,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          312,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      16,      16,     116,   0,   0
NCLNODE:                356,      0,       0,       0,       0,   0,   0
mqnode:                 344,      0,       3,       8,       3,   0,   0
mqueue:                 132,      0,       0,       0,       0,   0,   0
mvdata:                  32,      0,       0,       0,       0,   0,   0
mqnotifier:             136,      0,       0,       0,       0,   0,   0
procdesc:                72,      0,       0,       0,       0,   0,   0
Mountpoints:            672,      0,       3,      15,       3,   0,   0
pipe:                   408,      0,       1,      35,    1091,   0,   0
ksiginfo:                80,      0,      42,     158,     119,   0,   0
itimer:                 232,      0,       1,      16,       1,   0,   0
KNOTE:                   72,      0,       0,       0,      12,   0,   0
socket:                 424,   7668,      19,      35,    1202,   0,   0
ipq:                     32,    252,       0,       0,       0,   0,   0
udp_inpcb:              252,   7680,       8,      72,     160,   0,   0
udpcb:                    8,   7812,       8,     244,     160,   0,   0
tcp_inpcb:              252,   7680,       2,      62,       6,   0,   0
tcpcb:                  752,   7670,       2,      13,       6,   0,   0
tcptw:                   60,   1541,       0,       0,       0,   0,   0
syncache:               124,  15360,       0,       0,       0,   0,   0
hostcache:               76,  15370,       0,       0,       0,   0,   0
tcpreass:                20,    404,       0,       0,       0,   0,   0
sackhole:                20,      0,       0,       0,       0,   0,   0
sctp_ep:                976,   7668,       0,       0,       0,   0,   0
sctp_asoc:             1592,  40000,       0,       0,       0,   0,   0
sctp_laddr:              24,  80136,       0,       0,       3,   0,   0
sctp_raddr:             532,  80003,       0,       0,       0,   0,   0
sctp_chunk:              96, 400008,       0,       0,       0,   0,   0
sctp_readq:              76, 400044,       0,       0,       0,   0,   0
sctp_stream_msg_out:     68, 400020,       0,       0,       0,   0,   0
sctp_asconf:             24, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         24, 400008,       0,       0,       0,   0,   0
ripcb:                  252,   7680,       1,      63,      13,   0,   0
unpcb:                  172,   7682,       6,      63,    1013,   0,   0
rtentry:                108,      0,      13,      24,      14,   0,   0
IPFW dynamic rule:      108,   4107,       0,       0,       0,   0,   0
divcb:                  252,   7680,       0,       0,       0,   0,   0
selfd:                   28,      0,      31,     113,   13555,   0,   0
SWAPMETA:               276,  30086,      31,      39,      36,   0,   0
FFS inode:              116,      0,    2711,     111,    4398,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,    2711,      49,    4398,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq0: attimer0                    834262      10695
irq1: atkbd0                         275          3
irq7: ppc0                             1          0
irq11: pcm0 bge0                      92          1
irq14: ata0                        32354        414
irq15: ata1                          423          5
Total                             867407      11120

------------------------------------------------------------------------
pstat -T

 61/7663 files
0M/2047M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/label/Swap    4194040     1016  4193024     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
            ada0             ada1            pass0             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 29.37 407 11.69   3.50   1  0.00   0.00   0  0.00  57  2 13  0 27

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          616	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	0 packets sent
		0 data packets (0 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		90916420708437821 resends initiated by MTU discovery
		0 ack-only packets (0 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		0 control packets
	0 packets received
		0 acks (for 0 bytes)
		0 duplicate acks
		0 acks for unsent data
		0 packets (0 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	0 connection requests
	0 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	7525346126759879536 ignored RSTs in the windows
	0 connections established (including accepts)
	0 connections closed (including 4 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		14223757787205729232 connections updated cached ssthresh on close
	0 embryonic connections dropped
	0 segments updated rtt (of 0 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	0 correct data packet header predictions
	0 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		0 completed
		0 bucket overflow
		3231735672 cache overflow
		13878490090660978560 reset
		14024559434085171200 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	0 cookies sent
	0 cookies received
	0 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	8315177820320982133 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	39 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	2 with no checksum
	38 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	14202747855163872744 dropped due to full socket buffers
	0 not for hashed pcb
	4243996218545678873 delivered
	13876054981724591340 datagrams output
	90916420708441584 times multicast source filter matched
ip:
	40 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	13880185520411137644 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	39 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	0 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	13880189073412325376 packets sent from this host
	0 packets sent with fabricated ip header
	13880189107772063744 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	0 calls to icmp_error
	0 errors not generated in response to an icmp message
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		destination unreachable: 3311909552
		source quench: 3311909568
	0 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
ipsec:
	0 inbound packets violated process security policy
	0 inbound packets failed due to insufficient memory
	0 invalid inbound packets
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	8029953746877509737 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
ah:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets dropped; bad authentication detected
	0 packets dropped; bad authentication length
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
esp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 packets dropped; bad ilen
	0 replay counter wraps
	0 packets dropped; bad encryption detected
	0 packets dropped; bad authentication detected
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
	ESP output histogram:
		#139: 14027524820258542848
		#144: 2326202804928401481
		#148: 17042560347969784149
		#149: 615594868859046229
		#152: 13878550752779075852
		#153: 17042560347969784149
		#154: 7235444513182665806
ipcomp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	COMP output histogram:
		#6: 14027524820258542848
	0 packets sent uncompressed; size < compr. algo. threshold
	0 packets sent uncompressed; compression was useless
arp:
	2 ARP requests sent
	4 ARP replies sent
	5 ARP requests received
	0 ARP replies received
	0 ARP packets received
	7598817722882813545 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	0 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	0 packets for this host
	0 packets forwarded
	0 packets not forwardable
	4 redirects sent
	0 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Input histogram:
		#184: 13880216990699749376
		#185: 13880217025059487744
		#187: 219269010088473984
		#194: 14222497300200060400
		#195: 7233175079288729193
		#198: 14202724899545415680
		#233: 13879875595571065172
		#236: 17042560347969784149
		#239: 3263241216
		#240: 13880218897665228800
	Mbuf statistics:
		0 one mbuf
		two or more mbuf:
			(null)= 8029953746877509737
		0 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Input histogram:
		#11: 75153822012557581
		#15: 3231738164
		#16: 13878518420265265468
		#17: 13878517389473114436
		#18: 3231738188
		#19: 13878523986542881108
		#227: 13880216990699749376
		#228: 13880217025059487744
		#237: 14222505134140714960
		#240: 14202724645154207384
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	0 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	13878529449741283660 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
ipsec6:
	0 inbound packets violated process security policy
	0 inbound packets failed due to insufficient memory
	0 invalid inbound packets
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output
pfkey:
	0 requests sent from userland
	0 bytes sent from userland
	0 messages with invalid length field
	0 messages with invalid version field
	0 messages with invalid message type field
	0 messages too short
	0 messages with memory allocation failure
	0 messages with duplicate extension
	0 messages with invalid extension type
	0 messages with invalid sa type
	0 messages with invalid address extension
	0 requests sent to userland
	0 bytes sent to userland
	histogram by message type:
		#175: 7022364301717819465
		#182: 17042560347969784149
		#185: 13878548759914250504
		#186: 17042560347969784149
	0 messages toward single socket
	0 messages toward all sockets
	0 messages toward registered sockets
	0 messages with memory allocation failure

------------------------------------------------------------------------
netstat -m

513/399/912 mbufs in use (current/cache/total)
500/138/638/5014 mbuf clusters in use (current/cache/total/max)
512/137 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/2506 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/2226 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1668 16k jumbo clusters in use (current/cache/total/max)
1128K/375K/1504K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

------------------------------------------------------------------------
netstat -idW

Name      Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
bge0     1500 <Link#1>      00:10:18:0a:dd:27       45     0     0       41     0     0    0 
bge0     1500 192.168.1.0   192.168.1.184           39     -     -       38     -     -    - 
plip0*   1500 <Link#2>                               0     0     0        0     0     0    0 
lo0     16384 <Link#3>                               0     0     0        0     0     0    0 
lo0     16384 localhost     ::1                      0     -     -        0     -     -    - 
lo0     16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0     16384 your-net      localhost                0     -     -        0     -     -    - 
enc0*    1536 <Link#4>                               0     0     0        0     0     0    0 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0       32   bge0
127.0.0.1          link#3             UH          0        0    lo0
192.168.1.0/24     link#1             U           0        6   bge0
192.168.1.184      link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb    Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
c59822f0 tcp4       0      0 *.22               *.*                LISTEN
c59825e0 tcp6       0      0 *.22               *.*                LISTEN
c57bd4ec udp4       0      0 127.0.0.1.123      *.*                
c57bd5e8 udp6       0      0 fe80::1%lo0.123    *.*                
c57bd6e4 udp6       0      0 ::1.123            *.*                
c57bd7e0 udp4       0      0 192.168.1.184.123  *.*                
c57bd8dc udp6       0      0 *.123              *.*                
c57bd9d8 udp4       0      0 *.123              *.*                
c57bdad4 udp4       0      0 *.514              *.*                
c57bdbd0 udp6       0      0 *.514              *.*                
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c5940968 stream      0      0 c5941c34        0        0        0 /var/run/devd.pipe
c594060c dgram       0      0        0 c5940810        0 c59406b8
c59406b8 dgram       0      0        0 c5940810        0 c5940764
c5940764 dgram       0      0        0 c5940810        0        0
c5940810 dgram       0      0 c5968d50        0 c594060c        0 /var/run/logpriv
c59408bc dgram       0      0 c5968e6c        0        0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address         
tcp4  0/0/128        *.ssh                  
tcp6  0/0/128        *.ssh                  
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 6 at 0xffff
fstat: can't read file 7 at 0x78
fstat: can't read file 10 at 0xffffffff
fstat: can't read file 11 at 0x200007f
fstat: can't read file 12 at 0x1fffff
fstat: can't read file 13 at 0x4000000
fstat: can't read file 15 at 0xffff
fstat: can't read file 16 at 0x78
fstat: can't read file 19 at 0xffffffff
fstat: can't read file 20 at 0x200007f
fstat: can't read file 21 at 0x1fffff
fstat: can't read file 22 at 0x4000000
fstat: can't read file 24 at 0xffff
fstat: can't read file 25 at 0x78
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 6 at 0xffff
fstat: can't read file 7 at 0x78
fstat: can't read file 10 at 0xffffffff
fstat: can't read file 11 at 0x200007f
fstat: can't read file 12 at 0x1fffff
fstat: can't read file 13 at 0x4000000
fstat: can't read file 15 at 0xffff
fstat: can't read file 16 at 0x78
fstat: can't read file 19 at 0xffffffff
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 2 at 0x2000000
fstat: can't read file 4 at 0x4000000
fstat: can't read file 1 at 0xffffffff
fstat: can't read file 2 at 0x200007f
fstat: can't read file 3 at 0x1fffff
fstat: can't read file 4 at 0x4000000
fstat: can't read file 6 at 0xffff
fstat: can't read file 7 at 0x78
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     c++         1794 root /             2 drwxr-xr-x     512  r
root     c++         1794   wd /        1532195 drwxr-xr-x    1024  r
root     c++         1794 text /        323107 -r-xr-xr-x  19301616  r
root     c++         1794 ctty /dev         70 crw-------   ttyv1 rw
root     c++         1794    0 /dev         70 crw-------   ttyv1 rw
root     c++         1793 root /             2 drwxr-xr-x     512  r
root     c++         1793   wd /        1532195 drwxr-xr-x    1024  r
root     c++         1793 text /        323107 -r-xr-xr-x  19301616  r
root     c++         1793 ctty /dev         70 crw-------   ttyv1 rw
root     c++         1793    0 /dev         70 crw-------   ttyv1 rw
root     make        1725 root /             2 drwxr-xr-x     512  r
root     make        1725   wd /        1532195 drwxr-xr-x    1024  r
root     make        1725 text /        321734 -r-xr-xr-x  532776  r
root     make        1725 ctty /dev         70 crw-------   ttyv1 rw
root     make        1725    0 /dev         70 crw-------   ttyv1 rw
root     sh          1623 root /             2 drwxr-xr-x     512  r
root     sh          1623   wd /        809467 drwxr-xr-x     512  r
root     sh          1623 text /        3290533 -r-xr-xr-x  103076  r
root     sh          1623 ctty /dev         70 crw-------   ttyv1 rw
root     sh          1623    0 /dev         70 crw-------   ttyv1 rw
root     make        1619 root /             2 drwxr-xr-x     512  r
root     make        1619   wd /        1450151 drwxr-xr-x     512  r
root     make        1619 text /        321734 -r-xr-xr-x  532776  r
root     make        1619 ctty /dev         70 crw-------   ttyv1 rw
root     make        1619    0 /dev         70 crw-------   ttyv1 rw
root     make        1488 root /             2 drwxr-xr-x     512  r
root     make        1488   wd /        321026 drwxr-xr-x    1024  r
root     make        1488 text /        321734 -r-xr-xr-x  532776  r
root     make        1488 ctty /dev         70 crw-------   ttyv1 rw
root     make        1488    0 /dev         70 crw-------   ttyv1 rw
root     make        1483 root /             2 drwxr-xr-x     512  r
root     make        1483   wd /        321026 drwxr-xr-x    1024  r
root     make        1483 text /        321734 -r-xr-xr-x  532776  r
root     make        1483 ctty /dev         70 crw-------   ttyv1 rw
root     make        1483    0 /dev         70 crw-------   ttyv1 rw
root     csh         1479 root /             2 drwxr-xr-x     512  r
root     csh         1479   wd /        321026 drwxr-xr-x    1024  r
root     csh         1479 text /        3290501 -r-xr-xr-x  295816  r
root     csh         1479 ctty /dev         70 crw-------   ttyv1 rw
root     top         1478 root /             2 drwxr-xr-x     512  r
root     top         1478   wd /        642048 drwxr-xr-x     512  r
root     top         1478 text /        321074 -r-xr-xr-x   44520  r
root     top         1478 ctty /dev         71 crw-------   ttyv2 rw
root     top         1478    0 /dev         71 crw-------   ttyv2 rw
root     csh         1476 root /             2 drwxr-xr-x     512  r
root     csh         1476   wd /        642048 drwxr-xr-x     512  r
root     csh         1476 text /        3290501 -r-xr-xr-x  295816  r
root     csh         1476 ctty /dev         71 crw-------   ttyv2 rw
root     getty       1475 root /             2 drwxr-xr-x     512  r
root     getty       1475   wd /             2 drwxr-xr-x     512  r
root     getty       1475 text /        326373 -r-xr-xr-x   20816  r
root     getty       1475 ctty /dev         76 crw-------   ttyv7 rw
root     getty       1475    0 /dev         76 crw-------   ttyv7 rw
root     getty       1474 root /             2 drwxr-xr-x     512  r
root     getty       1474   wd /             2 drwxr-xr-x     512  r
root     getty       1474 text /        326373 -r-xr-xr-x   20816  r
root     getty       1474 ctty /dev         75 crw-------   ttyv6 rw
root     getty       1474    0 /dev         75 crw-------   ttyv6 rw
root     getty       1473 root /             2 drwxr-xr-x     512  r
root     getty       1473   wd /             2 drwxr-xr-x     512  r
root     getty       1473 text /        326373 -r-xr-xr-x   20816  r
root     getty       1473 ctty /dev         74 crw-------   ttyv5 rw
root     getty       1473    0 /dev         74 crw-------   ttyv5 rw
root     getty       1472 root /             2 drwxr-xr-x     512  r
root     getty       1472   wd /             2 drwxr-xr-x     512  r
root     getty       1472 text /        326373 -r-xr-xr-x   20816  r
root     getty       1472 ctty /dev         73 crw-------   ttyv4 rw
root     getty       1472    0 /dev         73 crw-------   ttyv4 rw
root     getty       1471 root /             2 drwxr-xr-x     512  r
root     getty       1471   wd /             2 drwxr-xr-x     512  r
root     getty       1471 text /        326373 -r-xr-xr-x   20816  r
root     getty       1471 ctty /dev         72 crw-------   ttyv3 rw
root     getty       1471    0 /dev         72 crw-------   ttyv3 rw
root     login       1470 root /             2 drwxr-xr-x     512  r
root     login       1470   wd /        642048 drwxr-xr-x     512  r
root     login       1470 text /        324375 -r-sr-xr-x   20220  r
root     login       1470 ctty /dev         71 crw-------   ttyv2 rw
root     login       1470    0 /dev         71 crw-------   ttyv2 rw
root     login       1469 root /             2 drwxr-xr-x     512  r
root     login       1469   wd /        642048 drwxr-xr-x     512  r
root     login       1469 text /        324375 -r-sr-xr-x   20220  r
root     login       1469 ctty /dev         70 crw-------   ttyv1 rw
root     login       1469    0 /dev         70 crw-------   ttyv1 rw
root     getty       1468 root /             2 drwxr-xr-x     512  r
root     getty       1468   wd /             2 drwxr-xr-x     512  r
root     getty       1468 text /        326373 -r-xr-xr-x   20816  r
root     getty       1468 ctty /dev         69 crw-------   ttyv0 rw
root     getty       1468    0 /dev         69 crw-------   ttyv0 rw
root     moused      1425 root /             2 drwxr-xr-x     512  r
root     moused      1425   wd /             2 drwxr-xr-x     512  r
root     moused      1425 text /        331879 -r-xr-xr-x   33784  r
root     moused      1425    0 /dev         25 crw-rw-rw-    null rw
root     cron        1402 root /             2 drwxr-xr-x     512  r
root     cron        1402   wd /        2327438 drwxr-x---     512  r
root     cron        1402 text /        329872 -r-xr-xr-x   32236  r
root     cron        1402    0 /dev         25 crw-rw-rw-    null rw
root     sshd        1398 root /             2 drwxr-xr-x     512  r
root     sshd        1398   wd /             2 drwxr-xr-x     512  r
root     sshd        1398 text /        325266 -r-xr-xr-x  239160  r
root     sshd        1398    0 /dev         25 crw-rw-rw-    null rw
root     ntpd        1369 root /             2 drwxr-xr-x     512  r
root     ntpd        1369   wd /             2 drwxr-xr-x     512  r
root     ntpd        1369 text /        322091 -r-xr-xr-x  324060  r
root     ntpd        1369    0 /dev         25 crw-rw-rw-    null rw
root     ntpd        1369    9 /dev         25 crw-rw-rw-    null rw
root     ntpd        1369   18 /dev         25 crw-rw-rw-    null rw
root     syslogd     1274 root /             2 drwxr-xr-x     512  r
root     syslogd     1274   wd /             2 drwxr-xr-x     512  r
root     syslogd     1274 text /        327708 -r-xr-xr-x   33220  r
root     syslogd     1274    0 /dev         25 crw-rw-rw-    null rw
root     syslogd     1274    9 /dev         25 crw-rw-rw-    null rw
root     syslogd     1274   18 /dev         25 crw-rw-rw-    null rw
root     devd        1127 root /             2 drwxr-xr-x     512  r
root     devd        1127   wd /             2 drwxr-xr-x     512  r
root     devd        1127 text /        3211443 -r-xr-xr-x  813828  r
root     devd        1127    0 /dev         25 crw-rw-rw-    null rw
_dhcp    dhclient    1126 root /        2327440 dr-xr-xr-x     512  r
_dhcp    dhclient    1126   wd /        2327440 dr-xr-xr-x     512  r
_dhcp    dhclient    1126 jail /        2327440 dr-xr-xr-x     512  r
_dhcp    dhclient    1126 text /        3211441 -r-xr-xr-x   73992  r
_dhcp    dhclient    1126    0 /dev         25 crw-rw-rw-    null rw
_dhcp    dhclient    1126    9 /dev         25 crw-rw-rw-    null rw
root     dhclient    1090 root /             2 drwxr-xr-x     512  r
root     dhclient    1090   wd /             2 drwxr-xr-x     512  r
root     dhclient    1090 text /        3211441 -r-xr-xr-x   73992  r
root     dhclient    1090    0 /dev         25 crw-rw-rw-    null rw
root     init           1 root /             2 drwxr-xr-x     512  r
root     init           1   wd /             2 drwxr-xr-x     512  r
root     init           1 text /        3211467 -r-xr-xr-x  757256  r
root     kernel         0 root /             2 drwxr-xr-x     512  r
root     kernel         0   wd /             2 drwxr-xr-x     512  r

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-PRERELEASE #1: Sat Jan 11 09:00:08 CET 2014
    root@music2.reztek:/usr/obj/usr/src/sys/MUSIC i386
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: VIA Samuel 2 (797.98-MHz 686-class CPU)
  Origin = "CentaurHauls"  Id = 0x673  Family = 0x6  Model = 0x7  Stepping = 3
  Features=0x803035<FPU,DE,TSC,MSR,MTRR,PGE,MMX>
  AMD Features=0x80000000<3DNow!>
real memory  = 268435456 (256 MB)
avail memory = 244457472 (233 MB)
random: <Software, Yarrow> initialized
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
apm0: <APM BIOS> on motherboard
apm0: found APM BIOS v1.2, connected at v1.2
pcib0 pcibus 0 on motherboard
pir0: <PCI Interrupt Routing Table: 8 Entries> on motherboard
pci0: <PCI bus> on pcib0
agp0: <VIA 8601 (Apollo ProMedia/PLE133Ta) host to PCI bridge> on hostb0
agp0: aperture size is 64M
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 0xdf000000-0xdf7fffff,0xdfae0000-0xdfafffff,0xde800000-0xdeffffff at device 0.0 on pci1
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <VIA 83C572 USB controller> port 0xec00-0xec1f irq 10 at device 7.3 on pci0
usbus0 on uhci0
viapropm0: <VIA VT82C686A Power Management Unit> port 0x400-0x40f at device 7.4 on pci0
smbus0: <System Management Bus> on viapropm0
smb0: <SMBus generic I/O> on smbus0
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x003001> mem 0xdfff0000-0xdfffffff irq 11 at device 11.0 on pci0
bge0: CHIP ID 0x00003001; ASIC REV 0x03; CHIP REV 0x30; PCI 33 MHz; 32bit
miibus0: <MII bus> on bge0
brgphy0: <BCM5705 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:10:18:0a:dd:27
pcm0: <Envy24HT audio (Generic)> port 0xe800-0xe81f,0xe400-0xe47f irq 11 at device 12.0 on pci0
pcm0: system configuration
  SubVendorID: 0x1412, SubDeviceID: 0x2403
  XIN2 Clock Source: 24.576MHz(96kHz*256)
  MPU-401 UART(s) #: not implemented
  ADC #: 1 and SPDIF receiver connected
  DAC #: 4
  Multi-track converter type: AC'97(SDATA_OUT:packed)
  S/PDIF(IN/OUT): 1/1 ID# 0x00
  GPIO(mask/dir/state): 0xff/0xff/0xff
cpu0 on motherboard
pmtimer0 on isa0
unknown: <PNP0a03> can't assign resources (port)
unknown: <PNP0c01> can't assign resources (memory)
attimer0: <AT timer> at port 0x40-0x43 irq 0 pnpid PNP0100 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> at port 0x70-0x71 irq 8 pnpid PNP0b00 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 irq 1 pnpid PNP0303 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 pnpid PNP0501 on isa0
uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 pnpid PNP0501 on isa0
ppc0: <ECP parallel printer port> at port 0x378-0x37f,0x778-0x77a irq 7 drq 3 pnpid PNP0401 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart2: <16550 or compatible> at port 0x3e8-0x3ef irq 5 pnpid PNP0501 on isa0
uart3: <16550 or compatible> at port 0x2e8-0x2ef irq 9 pnpid PNP0501 on isa0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcbfff,0xcc000-0xcd7ff,0xe0000-0xe0fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x180 on isa0
sc0: VGA <16 virtual consoles, flags=0x380>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0a03> can't assign resources (port)
unknown: <PNP0c01> can't assign resources (memory)
driver bug: Unable to set devclass (class: fdc devname: (unknown))
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny, logging disabled
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <VIA> at usbus0
uhub0: <VIA UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <Maxtor 93652U8 FA554E30> ATA-5 device
ada0: Serial Number M0000000
ada0: 66.700MB/s transfers (UDMA4, PIO 8192bytes)
ada0: 34837MB (71346240 512 byte sectors: 16H 255S/T 16383C)
ada0: Previously was known as ad0
ada1 at ata1 bus 0 scbus1 target 0 lun 0
ada1: <Transcend 128M 1.1> ATA-0 device
ada1: Serial Number SSSI128M04Z13A22431A
ada1: 16.700MB/s transfers (PIO4, PIO 512bytes)
ada1: 122MB (250368 512 byte sectors: 8H 32S/T 978C)
ada1: Previously was known as ad2
uhub0: 2 ports with 2 removable, self powered
Trying to mount root from ufs:/dev/label/Root [rw,noatime]...
WARNING: /rw/mnt was not properly dismounted
Setting hostuuid: 3ca7a290-710d-11e3-a608-00a0c94b3124.
Setting hostid: 0xfac708b2.
Entropy harvesting: interrupts ethernet point_to_point swi.
warning: total configured swap (524287 pages) exceeds maximum recommended amount (481376 pages).
warning: increase kern.maxswzone or reduce amount of swap.
Starting file system checks:
/dev/label/Root: DEFER FOR BACKGROUND CHECKING
WARNING: / was not properly dismounted
Mounting local file systems:.
Writing entropy file:.
Setting hostname: music2.reztek.
Starting Network: lo0 bge0 plip0 enc0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
	ether 00:10:18:0a:dd:27
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting devd.
Starting Network: plip0.
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting Network: enc0.
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting dhclient.
DHCPREQUEST on bge0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.184 -- renewal in 3600 seconds.
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Flushed all rules.
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
65000 allow ip from any to any
Firewall rules loaded.
Creating and/or trimming log files.
Starting syslogd.
No core dumps found.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Starting ntpd.
Configuring syscons: keyrate blanktime.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting default moused.
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
Starting background file system checks in 60 seconds.

Sun Feb  2 19:58:36 CET 2014
Feb  2 19:58:48 music2 login: ROOT LOGIN (root) ON ttyv2
Feb  2 19:59:02 music2 login: ROOT LOGIN (root) ON ttyv1
calcru: runtime went backwards from 102407 usec to 100464 usec for pid 1479 (csh)
calcru: runtime went backwards from 3221 usec to 3160 usec for pid 1479 (csh)
calcru: runtime went backwards from 123173 usec to 120836 usec for pid 1476 (csh)
calcru: runtime went backwards from 3236 usec to 3175 usec for pid 1476 (csh)
calcru: runtime went backwards from 13832 usec to 13569 usec for pid 1475 (getty)
calcru: runtime went backwards from 14486 usec to 14211 usec for pid 1474 (getty)
calcru: runtime went backwards from 13986 usec to 13721 usec for pid 1473 (getty)
calcru: runtime went backwards from 13541 usec to 13284 usec for pid 1472 (getty)
calcru: runtime went backwards from 13637 usec to 13378 usec for pid 1471 (getty)
calcru: runtime went backwards from 439925 usec to 431578 usec for pid 1470 (login)
calcru: runtime went backwards from 430411 usec to 422245 usec for pid 1469 (login)
calcru: runtime went backwards from 14666 usec to 14387 usec for pid 1468 (getty)
calcru: runtime went backwards from 6080 usec to 5965 usec for pid 1467 (sleep)
calcru: runtime went backwards from 8127 usec to 7973 usec for pid 1465 (logger)
calcru: runtime went backwards from 1916 usec to 1879 usec for pid 1464 (sh)
calcru: runtime went backwards from 960 usec to 941 usec for pid 1425 (moused)
calcru: runtime went backwards from 35678 usec to 35001 usec for pid 1402 (cron)
calcru: runtime went backwards from 129531 usec to 127073 usec for pid 1402 (cron)
calcru: runtime went backwards from 4575 usec to 4488 usec for pid 1398 (sshd)
calcru: runtime went backwards from 82237 usec to 81131 usec for pid 1369 (ntpd)
calcru: runtime went backwards from 50080 usec to 49244 usec for pid 1274 (syslogd)
calcru: runtime went backwards from 559 usec to 548 usec for pid 1127 (devd)
calcru: runtime went backwards from 2008 usec to 1970 usec for pid 1126 (dhclient)
calcru: runtime went backwards from 5698 usec to 5590 usec for pid 1090 (dhclient)
calcru: runtime went backwards from 261424 usec to 256463 usec for pid 1090 (dhclient)
calcru: runtime went backwards from 2864 usec to 2810 usec for pid 15 (usb)
calcru: runtime went backwards from 53852 usec to 53166 usec for pid 4 (cam)
calcru: runtime went backwards from 48080 usec to 47168 usec for pid 13 (geom)
calcru: runtime went backwards from 4733641 usec to 4644716 usec for pid 12 (intr)
calcru: runtime went backwards from 31166 usec to 30575 usec for pid 1 (init)
calcru: runtime went backwards from 14544175 usec to 14268208 usec for pid 1 (init)
calcru: runtime went backwards from 6105 usec to 5989 usec for pid 0 (kernel)
calcru: runtime went backwards from 1412020 usec to 1397733 usec for pid 1642 (cc)
calcru: runtime went backwards from 28811 usec to 28307 usec for pid 1641 (sh)
calcru: runtime went backwards from 165082 usec to 162195 usec for pid 1628 (make)
calcru: runtime went backwards from 745974 usec to 732927 usec for pid 1628 (make)
calcru: runtime went backwards from 12279 usec to 12065 usec for pid 1623 (sh)
calcru: runtime went backwards from 173321 usec to 170290 usec for pid 1623 (sh)
calcru: runtime went backwards from 117057 usec to 115010 usec for pid 1619 (make)
calcru: runtime went backwards from 74623 usec to 73318 usec for pid 1619 (make)
calcru: runtime went backwards from 158886 usec to 156107 usec for pid 1488 (make)
calcru: runtime went backwards from 4271263 usec to 4196562 usec for pid 1488 (make)
calcru: runtime went backwards from 33800 usec to 33052 usec for pid 1483 (make)
calcru: runtime went backwards from 75088 usec to 73425 usec for pid 1483 (make)
calcru: runtime went backwards from 110913 usec to 108457 usec for pid 1479 (csh)
calcru: runtime went backwards from 3160 usec to 3090 usec for pid 1479 (csh)
calcru: runtime went backwards from 120836 usec to 118161 usec for pid 1476 (csh)
calcru: runtime went backwards from 3175 usec to 3105 usec for pid 1476 (csh)
calcru: runtime went backwards from 13569 usec to 13269 usec for pid 1475 (getty)
calcru: runtime went backwards from 14211 usec to 13896 usec for pid 1474 (getty)
calcru: runtime went backwards from 13721 usec to 13417 usec for pid 1473 (getty)
calcru: runtime went backwards from 13284 usec to 12990 usec for pid 1472 (getty)
calcru: runtime went backwards from 13378 usec to 13082 usec for pid 1471 (getty)
calcru: runtime went backwards from 431578 usec to 422023 usec for pid 1470 (login)
calcru: runtime went backwards from 422245 usec to 412896 usec for pid 1469 (login)
calcru: runtime went backwards from 14387 usec to 14069 usec for pid 1468 (getty)
calcru: runtime went backwards from 7973 usec to 7797 usec for pid 1465 (logger)
calcru: runtime went backwards from 18177 usec to 17775 usec for pid 1464 (fsck)
calcru: runtime went backwards from 18473 usec to 18064 usec for pid 1464 (fsck)
calcru: runtime went backwards from 941 usec to 920 usec for pid 1425 (moused)
calcru: runtime went backwards from 53539 usec to 52603 usec for pid 1402 (cron)
calcru: runtime went backwards from 127073 usec to 124260 usec for pid 1402 (cron)
calcru: runtime went backwards from 4488 usec to 4388 usec for pid 1398 (sshd)
calcru: runtime went backwards from 94122 usec to 92825 usec for pid 1369 (ntpd)
calcru: runtime went backwards from 53106 usec to 51930 usec for pid 1274 (syslogd)
calcru: runtime went backwards from 598 usec to 585 usec for pid 1127 (devd)
calcru: runtime went backwards from 1970 usec to 1926 usec for pid 1126 (dhclient)
calcru: runtime went backwards from 5590 usec to 5466 usec for pid 1090 (dhclient)
calcru: runtime went backwards from 256463 usec to 250785 usec for pid 1090 (dhclient)
calcru: runtime went backwards from 7718 usec to 7639 usec for pid 19 (softdepflush)
calcru: runtime went backwards from 38789 usec to 38221 usec for pid 17 (syncer)
calcru: runtime went backwards from 358656 usec to 352431 usec for pid 16 (bufdaemon)
calcru: runtime went backwards from 4154 usec to 4081 usec for pid 15 (usb)
calcru: runtime went backwards from 47168 usec to 46123 usec for pid 13 (geom)
calcru: runtime went backwards from 5887433 usec to 5826978 usec for pid 12 (intr)
calcru: runtime went backwards from 92330108 usec to 91081619 usec for pid 11 (idle)
calcru: runtime went backwards from 30575 usec to 29898 usec for pid 1 (init)
calcru: runtime went backwards from 14268208 usec to 13952316 usec for pid 1 (init)
calcru: runtime went backwards from 7442 usec to 7312 usec for pid 0 (kernel)
panic: handle_written_inodeblock: Invalid link count 65529 for inodedep 0xc7648b00
KDB: stack backtrace:
#0 0xc0667f6f at kdb_backtrace+0x52
#1 0xc063dc7d at panic+0x85
#2 0xc084483f at softdep_disk_write_complete+0x1294
#3 0xc06b0656 at bufdone_finish+0x23
#4 0xc06b04f5 at bufdone+0x39
#5 0xc05e0d13 at g_vfs_done+0x289
#6 0xc05dd8ea at g_io_deliver+0x28f
#7 0xc05e0648 at g_std_done+0x4e
#8 0xc05dd8ea at g_io_deliver+0x28f
#9 0xc05e0648 at g_std_done+0x4e
#10 0xc05dd8ea at g_io_deliver+0x28f
#11 0xc05e0648 at g_std_done+0x4e
#12 0xc05dd8ea at g_io_deliver+0x28f
#13 0xc05db68f at g_disk_done_single+0xfa
#14 0xc0487fb7 at adadone+0x482
#15 0xc046de75 at xpt_done_process+0x435
#16 0xc0470a11 at xpt_done_td+0x15a
#17 0xc0615573 at fork_exit+0xa3
Uptime: 6m58s
Physical memory: 239 MB
Dumping 102 MB: 87 71 55 39 23 7

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	MUSIC
machine	i386
cpu	I686_CPU
cpu	I586_CPU
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	SC_TWOBUTTON_MOUSE
options	SC_HISTORY_SIZE=5000
options	SC_PIXEL_MODE
options	X86BIOS
options	VESA
options	ATA_STATIC_ID
options	DDB_CTF
options	KDB_TRACE
options	KDB
options	INCLUDE_CONFIG_FILE
options	KDTRACE_HOOKS
options	MAC
options	PROCDESC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	P1003_1B_MQUEUE
options	P1003_1B_SEMAPHORES
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	GEOM_LABEL
options	GEOM_PART_GPT
options	TMPFS
options	PSEUDOFS
options	PROCFS
options	NULLFS
options	FDESCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	KGSSAPI
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	TCP_SIGNATURE
options	IPSEC_NAT_T
options	IPSEC
options	LIBALIAS
options	IPDIVERT
options	IPFIREWALL_NAT
options	IPFIREWALL
options	SCTP
options	TCP_OFFLOAD
options	INET6
options	INET
options	PREEMPTION
options	SCHED_4BSD
options	NEW_PCIB
options	NATIVE
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
options	ISAPNP
device	isa
device	npx
device	mem
device	io
device	uart_ns8250
device	atpic
device	pci
device	agp
device	fdc
device	ata
device	scbus
device	da
device	sa
device	cd
device	pass
device	ses
device	ctl
device	atkbdc
device	atkbd
device	kbdmux
device	psm
device	vga
device	splash
device	sc
device	apm
device	pmtimer
device	smbus
device	viapm
device	smb
device	iicbus
device	iicbb
device	iicsmb
device	uart
device	ppc
device	ppbus
device	lpt
device	plip
device	ppi
device	mii
device	brgphy
device	bge
device	loop
device	random
device	ether
device	vlan
device	enc
device	tun
device	gif
device	faith
device	crypto
device	pty
device	md
device	firmware
device	bpf
device	uhci
device	usb
device	uhid
device	ukbd
device	umass
device	ums
device	sound
device	snd_envy24ht
device	snd_spicds

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140202204623.00003fe5>