Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 10 Jul 2006 12:55:13 +0300
From:      Nikolay Pavlov <quetzal@zone3000.net>
To:        freebsd-stable@freebsd.org
Subject:   Re: Processes in block state in vmstat.
Message-ID:  <20060710095513.GA4471@zone3000.net>
In-Reply-To: <20060707134026.GA1750@zone3000.net>
References:  <20060707134026.GA1750@zone3000.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On Friday,  7 July 2006 at 16:40:26 +0300, Nikolay Pavlov wrote:
> Hi, folks.
> I want to add some additional information about problem related
> processes in block state. I see it on my server Dual CPU: Intel(R)
> Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device.

It seems my issue could be related this PR
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/94139
i have submitted follow-up report with backtraces.

> 
> FreeBSD 6.1-RELEASE-p1
> 
> I see some httpd daemons in D state:
> 
> procs      memory      page                   disk   faults      cpu
>  r b w     avm    fre  flt  re  pi  po  fr  sr am0   in   sy  cs us sy
> id
>  0 527 0 2173832  94028   49   0   0   0 2722 2658   0 9219 4140 17350 1  7 92
>  0 528 0 2172836  93636   66   0   0   0 218   0   6  504  512 1221  0 2 98
>  0 523 0 2171828  93360   56   0   0   0 188   0   6  551  298 1286  0 2 98
>  0 523 0 2171972  92612   29   0   0   0 189   0   6  520  395 1250  0 2 98
> 30 494 0 2170976  91836   92   0   0   0 333   0  11  448  250 970  0  2 98
>  0 518 0 2171156  90580   43   1   0   0 314   0  10  640  758 1676  0 3 97
>  0 510 0 2170244  89452   61   0   0   0 402   0  14  615  644 1667  0 3 97
>  1 512 0 2170420  89292    1   0   0   0   0   0   0  493  307 1133  0 2 98
>  0 517 0 2169560  86868  110   0   0   0 735   0  27  623  852 1584  0 3 96
>  0 514 0 2168552  87176   53   0   0   0 198   0   0  487  200 1078  0 1 99
>  0 517 0 2178752  84040 1523   0   0   0 1261   0   0  512 2660 1320  2 3 96
> 
> vmstat -s
> 
> 3129790854 cpu context switches
> 1663101011 device interrupts
>   5504551 software interrupts
>   9321603 traps
> 746764672 system calls
>        54 kernel threads created
>     66887  fork() calls
>      9669 vfork() calls
>         0 rfork() calls
>      4784 swap pager pageins
>      5635 swap pager pages paged in
>      1902 swap pager pageouts
>      3590 swap pager pages paged out
>     24668 vnode pager pageins
>     64050 vnode pager pages paged in
>         0 vnode pager pageouts
>         0 vnode pager pages paged out
>     28140 page daemon wakeups
> 479450699 pages examined by the page daemon
>     75270 pages reactivated
>   2461680 copy-on-write faults
>      3805 copy-on-write optimized faults
>   4531914 zero fill pages zeroed
>   2732630 zero fill pages prezeroed
>     18095 intransit blocking page faults
>   8922821 total VM faults taken
>         0 pages affected by kernel thread creation
>   8018826 pages affected by  fork()
>   1107169 pages affected by vfork()
>         0 pages affected by rfork()
> 491039188 pages freed
>       393 pages freed by daemon
>   5744590 pages freed by exiting processes
>    201969 pages active
>    184930 pages inactive
>     17190 pages in VM cache
>    107624 pages wired down
>       750 pages free
>     4096 bytes per page
>  30979017 total name lookups
>           cache hits (82% pos + 1% neg) system 1% per-directory
>           deletions 0%, falsehits 0%, toolong 0%
> 
> 
> vmstat -z
> 
> ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
> 
> UMA Kegs:        140,        0,      83,     13,       83
> UMA Zones:       480,        0,      83,      5,       83
> UMA Slabs:        64,        0,    2537,    177,   225611
> UMA RCntSlabs:   104,        0,   32769,     13,  2949442
> UMA Hash:        128,        0,       3,     27,        6
> 16 Bucket:        76,        0,      27,     23,       73
> 32 Bucket:       140,        0,      23,     33,       64
> 64 Bucket:       268,        0,      17,     39,      100
> 128 Bucket:      524,        0,     223,     71,     1036
> VM OBJECT:       132,        0,   30655,   2318,  1833813
> MAP:             192,        0,       7,     33,        7
> KMAP ENTRY:       68,    65520,     285,  11251, 78210624
> MAP ENTRY:        68,        0,  215821,   8011,  3882740
> PV ENTRY:         24,  2155135, 1391603, 119877, 42095182
> DP fakepg:        72,        0,       0,     53,        6
> mt_zone:        1024,        0,     174,    126,      174
> 16:               16,        0,    2158,    684,   578060
> 32:               32,        0,    3343,    386,    87211
> 64:               64,        0,    7323,    819,  1449967
> 128:             128,        0,    3507,    333,   519528
> 256:             256,        0,    6226,    509,   181376
> 512:             512,        0,      64,     96,    77544
> 1024:           1024,        0,      55,     73,    38283
> 2048:           2048,        0,     141,     27,    25379
> 4096:           4096,        0,    1990,    135,    84846
> Files:            72,        0,    2924,   1475,  1665330
> PROC:            524,        0,    1936,    220,    76612
> THREAD:          372,        0,    2156,     14,     2156
> KSEGRP:           88,        0,    2156,     84,     2156
> UPCALL:           44,        0,       0,      0,        0
> VMSPACE:         300,        0,    1886,    337,    76512
> mbuf_packet:     256,        0,   66135,      0, 1979970489
> mbuf:            256,        0,       7,   1253, 3511131461
> mbuf_cluster:   2048,    65536,   65538,      0, 1966671403
> mbuf_jumbo_pagesize: 4096,        0,       0,      0,        0
> mbuf_jumbo_9k:  9216,        0,       0,      0,        0
> mbuf_jumbo_16k: 16384,        0,       0,      0,        0
> ACL UMA zone:    388,        0,       0,      0,        0
> g_bio:           132,        0,       0,     87, 79264695
> ata_request:     204,        0,       0,      0,        0
> ata_composite:   196,        0,       0,      0,        0
> VNODE:           272,        0,   27508,   7072,  4868108
> VNODEPOLL:        76,        0,       1,     49,        1
> S VFS Cache:      68,        0,   27595,   2869,  4582987
> L VFS Cache:     291,        0,    1708,    203,   314868
> NAMEI:          1024,        0,       3,    129, 12912949
> DIRHASH:        1024,        0,     495,     41,     4186
> NFSMOUNT:        480,        0,       2,     14,        3
> NFSNODE:         460,        0,      13,     27,     1525
> PIPE:            408,        0,      28,     35,    31594
> KNOTE:            68,        0,       0,    168,     5762
> socket:          356,    16434,    2327,   1699,   277319
> unpcb:           140,    65548,      42,     70,    12935
> ipq:              32,     2147,       0,      0,     2235
> udpcb:           180,    16434,       5,    127,     5531
> inpcb:           180,    16434,    2259,   1393,   258840
> tcpcb:           464,    16424,    2258,   1454,   258840
> tcptw:            48,     3354,       1,    389,    38232
> syncache:        100,    15366,      17,    334,   322283
> hostcache:        76,    15400,     246,    304,     6938
> tcpreass:         20,     4225,       1,    168,     4936
> sackhole:         20,        0,       8,    499, 23975962
> ripcb:           180,    16434,       0,     44,       11
> rtentry:         132,        0,       5,     53,       54
> pfsrctrpl:       100,        0,       0,      0,        0
> pfrulepl:        604,        0,       0,      0,        0
> pfstatepl:       260,    10005,       0,      0,        0
> pfaltqpl:        128,        0,       0,      0,        0
> pfpooladdrpl:     68,        0,       0,      0,        0
> pfrktable:      1240,        0,       0,      0,        0
> pfrkentry:       156,        0,       0,      0,        0
> pfrkentry2:      156,        0,       0,      0,        0
> pffrent:          16,     5075,       0,      0,        0
> pffrag:           48,        0,       0,      0,        0
> pffrcache:        48,    10062,       0,      0,        0
> pffrcent:         12,    50141,       0,      0,        0
> pfstatescrub:     28,        0,       0,      0,        0
> pfiaddrpl:        92,        0,       0,      0,        0
> pfospfen:        108,        0,       0,      0,        0
> pfosfp:           28,        0,       0,      0,        0
> IPFW dynamic rule zone:  108,        0,       0,      0,        0
> SWAPMETA:        276,   121576,      36,     20,     1613
> FFS inode:       132,        0,   27459,   1280,  4866112
> FFS1 dinode:     128,        0,       0,      0,        0
> FFS2 dinode:     256,        0,   27459,   1206,  4866112
> 
> Intresting thing that everytime i use "vmstat 1" problem gone
> and i save my uptime.
> 
> Server is still under load and i see only one process in D state 
> 
> 85815  ??  D      0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0
> 
> Also i have found that it's much easy to trigger this state for httpd
> when i use low (default) value for MaxSpareServers (I have about 
> 300 ~ 500 connections all the time)
> 
> 
> -- 
> ========================================================================= 
> = Best regards, Nikolay Pavlov. <<<------------------------------------ = 
> ========================================================================= 
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

-- 
========================================================================= 
= Best regards, Nikolay Pavlov. <<<------------------------------------ = 
========================================================================= 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060710095513.GA4471>