Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 28 Jan 2011 13:41:01 +0100
From:      Bartosz Stec <bartosz.stec@it4pro.pl>
To:        Damien Fleuriot <ml@my.gd>
Cc:        freebsd-stable@freebsd.org
Subject:   Re: top shows only part of available physmem
Message-ID:  <4D42B95D.6020503@it4pro.pl>
In-Reply-To: <4D42A8EF.7060302@my.gd>
References:  <4D401192.3030400@it4pro.pl>	<201101261235.56856.jhb@freebsd.org>	<20110126180402.GA17271@tolstoy.tols.org>	<201101261344.50756.jhb@freebsd.org>	<4D40C355.6070306@it4pro.pl>	<20110127032142.GA19946@icarus.home.lan>	<4D417931.1060009@it4pro.pl> <4D429C71.6000100@it4pro.pl> <4D42A8EF.7060302@my.gd>

next in thread | previous in thread | raw e-mail | index | archive | help
W dniu 2011-01-28 12:30, Damien Fleuriot pisze:
>
> On 1/28/11 11:37 AM, Bartosz Stec wrote:
>>>>>>>>>>> Guys,
>>>>>>>>>>>
>>>>>>>>>>> could someone explain me this?
>>>>>>>>>>>
>>>>>>>>>>>        # sysctl hw.realmem
>>>>>>>>>>>        hw.realmem: 2139029504
>>>>>>>>>>>
>>>>>>>>>>> top line shows:
>>>>>>>>>>>
>>>>>>>>>>>        Mem: 32M Active, 35M Inact, 899M Wired, 8392K Cache,
>>>>>>>>>>> 199M Buf, 58M Free
>>>>>>>>>>>
>>>>>>>>>>> 32+35+899+8+199+58 = 1231MB
>>>>>>>>>>>
>>>>>>>>>>> Shouldn't that sum to all available ram? Or maybe I'm reading
>>>>>>>>>>> it wrong?
>>>>>>>>>>> This machine has indeed 2GB of ram on board and showed in BIOS.
>>>>>>>>>>> i386  FreeBSD 8.2-PRERELEASE #16: Mon Jan 17 22:28:53 CET 2011
>>>>>>>>>>> Cheers.
>>>>>>>>>> First, don't include 'buf' as isn't a separate set of RAM, it
>>>>>>>>>> is only a range
>>>>>>>>>> of the virtual address space in the kernel.  It used to be
>>>>>>>>>> relevant when the
>>>>>>>>>> buffer cache was separate from the VM page cache, but now it is
>>>>>>>>>> mostly
>>>>>>>>>> irrelevant (arguably it should just be dropped from top output).
>>>>>>>>> Thanks for the explanation. So 1231MB - 199MB Buf and we got
>>>>>>>>> about 1GB
>>>>>>>>> of memory instead of 2B.
>>>>>>>>>
>>>>>>>>>> However, look at what hw.physmem says (and the realmem and
>>>>>>>>>> availmem lines in
>>>>>>>>>> dmesg).  realmem is actually not that useful as it is not a
>>>>>>>>>> count of the
>>>>>>>>>> amount of memory, but the address of the highest memory page
>>>>>>>>>> available.  There
>>>>>>>>>> can be less memory available than that due to "holes" in the
>>>>>>>>>> address space for
>>>>>>>>>> PCI memory BARs, etc.
>>>>>>>>>>
>>>>>>>>> OK, here you go:
>>>>>>>>> # sysctl hw | grep mem
>>>>>>>>>
>>>>>>>>>       hw.physmem: 2125893632
>>>>>>>>>       hw.usermem: 1212100608
>>>>>>>>>       hw.realmem: 2139029504
>>>>>>>>>       hw.pci.host_mem_start: 2147483648
>>>>>>>> Humm, you should still have 2GB of RAM then.  All the memory you
>>>>>>>> set aside
>>>>>>>> for ARC should be counted in the 'wired' count, so I'm not sure
>>>>>>>> why you see
>>>>>>>> 1GB of RAM rather than 2GB.
>>>>>>> For what its worth (seems to be the same values top shows), the
>>>>>>> sysctl's
>>>>>>> I use to make cacti graphs of memory usage are: (Counts are in pages)
>>>>>>>
>>>>>>> vm.stats.vm.v_page_size
>>>>>>>
>>>>>>> vm.stats.vm.v_wire_count
>>>>>>> vm.stats.vm.v_active_count
>>>>>>> vm.stats.vm.v_inactive_count
>>>>>>> vm.stats.vm.v_cache_count
>>>>>>> vm.stats.vm.v_free_count
>>>>>>>
>>>>>>> Using the output of those sysctls I allways get a cacti graph
>>>>>>> which at
>>>>>>> least very much seems to account for all memory, and has a flat
>>>>>>> surface
>>>>>>> in a stacked graph.
>>>>>> These sysctls are exactly what top uses.  There is also a
>>>>>> 'v_page_count'
>>>>>> which is a total count of pages.
>>>>>>
>>>>> So here's additional sysctl output from now:
>>>>>
>>>>>      fbsd# sysctl hw | grep mem
>>>>>      hw.physmem: 2125893632
>>>>>      hw.usermem: 1392594944
>>>>>      hw.realmem: 2139029504
>>>>>      hw.pci.host_mem_start: 2147483648
>>>>>
>>>>>      fbsd# sysctl vm.stats.vm
>>>>>      vm.stats.vm.v_kthreadpages: 0
>>>>>      vm.stats.vm.v_rforkpages: 0
>>>>>      vm.stats.vm.v_vforkpages: 1422927
>>>>>      vm.stats.vm.v_forkpages: 4606557
>>>>>      vm.stats.vm.v_kthreads: 40
>>>>>      vm.stats.vm.v_rforks: 0
>>>>>      vm.stats.vm.v_vforks: 9917
>>>>>      vm.stats.vm.v_forks: 30429
>>>>>      vm.stats.vm.v_interrupt_free_min: 2
>>>>>      vm.stats.vm.v_pageout_free_min: 34
>>>>>      vm.stats.vm.v_cache_max: 27506
>>>>>      vm.stats.vm.v_cache_min: 13753
>>>>>      vm.stats.vm.v_cache_count: 20312
>>>>>      vm.stats.vm.v_inactive_count: 18591
>>>>>      vm.stats.vm.v_inactive_target: 20629
>>>>>      vm.stats.vm.v_active_count: 1096
>>>>>      vm.stats.vm.v_wire_count: 179027
>>>>>      vm.stats.vm.v_free_count: 6193
>>>>>      vm.stats.vm.v_free_min: 3260
>>>>>      vm.stats.vm.v_free_target: 13753
>>>>>      vm.stats.vm.v_free_reserved: 713
>>>>>      vm.stats.vm.v_page_count: 509752
>>>>>      vm.stats.vm.v_page_size: 4096
>>>>>      vm.stats.vm.v_tfree: 196418851
>>>>>      vm.stats.vm.v_pfree: 2837177
>>>>>      vm.stats.vm.v_dfree: 0
>>>>>      vm.stats.vm.v_tcached: 1305893
>>>>>      vm.stats.vm.v_pdpages: 3527455
>>>>>      vm.stats.vm.v_pdwakeups: 187
>>>>>      vm.stats.vm.v_reactivated: 83786
>>>>>      vm.stats.vm.v_intrans: 3053
>>>>>      vm.stats.vm.v_vnodepgsout: 134384
>>>>>      vm.stats.vm.v_vnodepgsin: 29213
>>>>>      vm.stats.vm.v_vnodeout: 96249
>>>>>      vm.stats.vm.v_vnodein: 29213
>>>>>      vm.stats.vm.v_swappgsout: 19730
>>>>>      vm.stats.vm.v_swappgsin: 8573
>>>>>      vm.stats.vm.v_swapout: 5287
>>>>>      vm.stats.vm.v_swapin: 2975
>>>>>      vm.stats.vm.v_ozfod: 83338
>>>>>      vm.stats.vm.v_zfod: 2462557
>>>>>      vm.stats.vm.v_cow_optim: 330
>>>>>      vm.stats.vm.v_cow_faults: 1239253
>>>>>      vm.stats.vm.v_vm_faults: 5898471
>>>>>
>>>>>      fbsd# sysctl vm.vmtotal
>>>>>      vm.vmtotal:
>>>>>      System wide totals computed every five seconds: (values in
>>>>> kilobytes)
>>>>>      ===============================================
>>>>>      Processes:              (RUNQ: 1 Disk Wait: 0 Page Wait: 0
>>>>> Sleep: 60)
>>>>>      Virtual Memory:         (Total: 4971660K Active: 699312K)
>>>>>      Real Memory:            (Total: 540776K Active: 29756K)
>>>>>      Shared Virtual Memory:  (Total: 41148K Active: 19468K)
>>>>>      Shared Real Memory:     (Total: 4964K Active: 3048K)
>>>>>      Free Memory Pages:      105308K
>>>>>
>>>>>
>>>>>      /usr/bin/top line: Mem: 4664K Active, 73M Inact, 700M Wired, 79M
>>>>>      Cache, 199M Buf, 23M Free
>>>>>      Sum (Without Buf): 879,5 MB
>>>>>
>>>>>      So what are we looking at? Wrong sysctls/top output or maybe
>>>>>      actually FreeBSD doesn't use all available RAM for some reason?
>>>>>      Could it be hardware problem? Maybe I should provide some
>>>>> additional
>>>>>      data?
>>>> Does the behaviour become more expected if you remove ZFS from the
>>>> picture?  Please try this (yes really).
>>>>
>>> About an hour ago I had to hard reset this machine because it stopped
>>> responding (bu still gived ping response) after massive slowdown seen
>>> by SAMBA users.
>>> Now top shows following:
>>> Mem: 78M Active, 83M Inact, 639M Wired, 120K Cache, 199M Buf, 1139M Free.
>>>
>>> What I am afraid is that this PC slowly eats own memory and finally
>>> starved itself to death, because it happened second time in 2 weeks,
>>> and it seems that rebuilding world+kernel Mon Jan 17 22:28:53 CET 2011
>>> could be the cause. For some strange reason I believe that Jeremy
>>> Chadwick could be right pointing ZFS. Way this machine stops
>>> responding without any info in logs makes me believe that it has
>>> simply lost ability to I/O to HDD (system is ZFS-only).
>>>
>> Day 2 after reboot:
>> Mem: 100M Active, 415M Inact, 969M Wired, 83M Cache, 199M Buf, 21M Free
>> Sum: 1588MB
>> 1/4 of total RAM disappeared already.
>> Anyone knows what possibly happening here or maybe I should hire some
>> voodoo shaman to expel memory-eating-ghost from the machine ;)?
>>
>
> Can you provide the following sysctls (ignore my values obviously)
> again, now that some of your memory magicked itself away ?
>
> hw.physmem: 4243976192
> hw.usermem: 3417485312
> hw.realmem: 5100273664
> vfs.zfs.arc_min: 134217728
> vfs.zfs.arc_max: 2147483648
>
>
> And check out the ZFS ARC stats script here:
> http://bitbucket.org/koie/arc_summary/changeset/dbe14d2cf52b/
>
> Run it and see what results you get concerning your ZFS used memory.
> What's of interest is the current size of your ZFS ARC cache.
> It might account for the memory you're missing, with a bit of luck.
Sure:

    hw.physmem: 2125893632
    hw.usermem: 898928640
    hw.realmem: 2139029504
    vfs.zfs.arc_min: 167772160
    vfs.zfs.arc_max: 1342177280


Actual top stats:

    53M Active, 145M Inact, 1175M Wired, 68M Cache, 199M Buf, 7716K Free

Sum: 1448MB (without Buf)

About 150MB less than 2 hours ago ;)

    # ./arc_summary.sh
    System Memory:
              Physical RAM:  2027 MB
              Free Memory :  7 MB

    ARC Size:
              Current Size:             796 MB (arcsize)
              Target Size (Adaptive):   797 MB (c)
              Min Size (Hard Limit):    160 MB (zfs_arc_min)
              Max Size (Hard Limit):    1280 MB (zfs_arc_max)

    ARC Size Breakdown:
              Most Recently Used Cache Size:          52%    415 MB (p)
              Most Frequently Used Cache Size:        47%    382 MB (c-p)

    ARC Efficency:
              Cache Access Total:             5931999
              Cache Hit Ratio:      89%       5323807        [Defined
    State for buffer]
              Cache Miss Ratio:     10%       608192         [Undefined
    State for Buffer]
              REAL Hit Ratio:       89%       5317666        [MRU/MFU
    Hits Only]

              Data Demand   Efficiency:    95%
              Data Prefetch Efficiency:     1%

             CACHE HITS BY CACHE LIST:
               Anon:                       --%        Counter Rolled.
               Most Recently Used:         39%        2121911
    (mru)          [ Return Customer ]
               Most Frequently Used:       60%        3195755
    (mfu)          [ Frequent Customer ]
               Most Recently Used Ghost:    1%        56946
    (mru_ghost)      [ Return Customer Evicted, Now Back ]
               Most Frequently Used Ghost:  3%        175154
    (mfu_ghost)     [ Frequent Customer Evicted, Now Back ]
             CACHE HITS BY DATA TYPE:
               Demand Data:                21%        1164556
               Prefetch Data:               0%        188
               Demand Metadata:            77%        4151758
               Prefetch Metadata:           0%        7305
             CACHE MISSES BY DATA TYPE:
               Demand Data:                 9%        59296
               Prefetch Data:               2%        15143
               Demand Metadata:            85%        518463
               Prefetch Metadata:           2%        15290
    ---------------------------------------------

Tunables in loader.conf:

    vm.kmem_size="1536M"
    vm.kmem_size_max="1536M"
    vfs.zfs.arc_max="1280M"


It seems that about 579MB is now "missing" while ARC size is 796 MB so 
it's rather not the case.

-- 
Bartosz Stec





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D42B95D.6020503>