From owner-freebsd-stable@FreeBSD.ORG Mon Jul 10 09:55:44 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22F0D16A4E6 for ; Mon, 10 Jul 2006 09:55:44 +0000 (UTC) (envelope-from quetzal@zone3000.net) Received: from mx1.sitevalley.com (sitevalley.com [209.67.60.43]) by mx1.FreeBSD.org (Postfix) with SMTP id 9F8DA43D45 for ; Mon, 10 Jul 2006 09:55:43 +0000 (GMT) (envelope-from quetzal@zone3000.net) Received: from zone3000.kharkov.ua (HELO localhost) (217.144.68.98) by 209.67.61.254 with SMTP; 10 Jul 2006 09:55:41 -0000 Date: Mon, 10 Jul 2006 12:55:13 +0300 From: Nikolay Pavlov To: freebsd-stable@freebsd.org Message-ID: <20060710095513.GA4471@zone3000.net> Mail-Followup-To: Nikolay Pavlov , freebsd-stable@freebsd.org References: <20060707134026.GA1750@zone3000.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060707134026.GA1750@zone3000.net> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 6.1-RELEASE Subject: Re: Processes in block state in vmstat. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jul 2006 09:55:44 -0000 On Friday, 7 July 2006 at 16:40:26 +0300, Nikolay Pavlov wrote: > Hi, folks. > I want to add some additional information about problem related > processes in block state. I see it on my server Dual CPU: Intel(R) > Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device. It seems my issue could be related this PR http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/94139 i have submitted follow-up report with backtraces. > > FreeBSD 6.1-RELEASE-p1 > > I see some httpd daemons in D state: > > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr am0 in sy cs us sy > id > 0 527 0 2173832 94028 49 0 0 0 2722 2658 0 9219 4140 17350 1 7 92 > 0 528 0 2172836 93636 66 0 0 0 218 0 6 504 512 1221 0 2 98 > 0 523 0 2171828 93360 56 0 0 0 188 0 6 551 298 1286 0 2 98 > 0 523 0 2171972 92612 29 0 0 0 189 0 6 520 395 1250 0 2 98 > 30 494 0 2170976 91836 92 0 0 0 333 0 11 448 250 970 0 2 98 > 0 518 0 2171156 90580 43 1 0 0 314 0 10 640 758 1676 0 3 97 > 0 510 0 2170244 89452 61 0 0 0 402 0 14 615 644 1667 0 3 97 > 1 512 0 2170420 89292 1 0 0 0 0 0 0 493 307 1133 0 2 98 > 0 517 0 2169560 86868 110 0 0 0 735 0 27 623 852 1584 0 3 96 > 0 514 0 2168552 87176 53 0 0 0 198 0 0 487 200 1078 0 1 99 > 0 517 0 2178752 84040 1523 0 0 0 1261 0 0 512 2660 1320 2 3 96 > > vmstat -s > > 3129790854 cpu context switches > 1663101011 device interrupts > 5504551 software interrupts > 9321603 traps > 746764672 system calls > 54 kernel threads created > 66887 fork() calls > 9669 vfork() calls > 0 rfork() calls > 4784 swap pager pageins > 5635 swap pager pages paged in > 1902 swap pager pageouts > 3590 swap pager pages paged out > 24668 vnode pager pageins > 64050 vnode pager pages paged in > 0 vnode pager pageouts > 0 vnode pager pages paged out > 28140 page daemon wakeups > 479450699 pages examined by the page daemon > 75270 pages reactivated > 2461680 copy-on-write faults > 3805 copy-on-write optimized faults > 4531914 zero fill pages zeroed > 2732630 zero fill pages prezeroed > 18095 intransit blocking page faults > 8922821 total VM faults taken > 0 pages affected by kernel thread creation > 8018826 pages affected by fork() > 1107169 pages affected by vfork() > 0 pages affected by rfork() > 491039188 pages freed > 393 pages freed by daemon > 5744590 pages freed by exiting processes > 201969 pages active > 184930 pages inactive > 17190 pages in VM cache > 107624 pages wired down > 750 pages free > 4096 bytes per page > 30979017 total name lookups > cache hits (82% pos + 1% neg) system 1% per-directory > deletions 0%, falsehits 0%, toolong 0% > > > vmstat -z > > ITEM SIZE LIMIT USED FREE REQUESTS > > UMA Kegs: 140, 0, 83, 13, 83 > UMA Zones: 480, 0, 83, 5, 83 > UMA Slabs: 64, 0, 2537, 177, 225611 > UMA RCntSlabs: 104, 0, 32769, 13, 2949442 > UMA Hash: 128, 0, 3, 27, 6 > 16 Bucket: 76, 0, 27, 23, 73 > 32 Bucket: 140, 0, 23, 33, 64 > 64 Bucket: 268, 0, 17, 39, 100 > 128 Bucket: 524, 0, 223, 71, 1036 > VM OBJECT: 132, 0, 30655, 2318, 1833813 > MAP: 192, 0, 7, 33, 7 > KMAP ENTRY: 68, 65520, 285, 11251, 78210624 > MAP ENTRY: 68, 0, 215821, 8011, 3882740 > PV ENTRY: 24, 2155135, 1391603, 119877, 42095182 > DP fakepg: 72, 0, 0, 53, 6 > mt_zone: 1024, 0, 174, 126, 174 > 16: 16, 0, 2158, 684, 578060 > 32: 32, 0, 3343, 386, 87211 > 64: 64, 0, 7323, 819, 1449967 > 128: 128, 0, 3507, 333, 519528 > 256: 256, 0, 6226, 509, 181376 > 512: 512, 0, 64, 96, 77544 > 1024: 1024, 0, 55, 73, 38283 > 2048: 2048, 0, 141, 27, 25379 > 4096: 4096, 0, 1990, 135, 84846 > Files: 72, 0, 2924, 1475, 1665330 > PROC: 524, 0, 1936, 220, 76612 > THREAD: 372, 0, 2156, 14, 2156 > KSEGRP: 88, 0, 2156, 84, 2156 > UPCALL: 44, 0, 0, 0, 0 > VMSPACE: 300, 0, 1886, 337, 76512 > mbuf_packet: 256, 0, 66135, 0, 1979970489 > mbuf: 256, 0, 7, 1253, 3511131461 > mbuf_cluster: 2048, 65536, 65538, 0, 1966671403 > mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0 > mbuf_jumbo_9k: 9216, 0, 0, 0, 0 > mbuf_jumbo_16k: 16384, 0, 0, 0, 0 > ACL UMA zone: 388, 0, 0, 0, 0 > g_bio: 132, 0, 0, 87, 79264695 > ata_request: 204, 0, 0, 0, 0 > ata_composite: 196, 0, 0, 0, 0 > VNODE: 272, 0, 27508, 7072, 4868108 > VNODEPOLL: 76, 0, 1, 49, 1 > S VFS Cache: 68, 0, 27595, 2869, 4582987 > L VFS Cache: 291, 0, 1708, 203, 314868 > NAMEI: 1024, 0, 3, 129, 12912949 > DIRHASH: 1024, 0, 495, 41, 4186 > NFSMOUNT: 480, 0, 2, 14, 3 > NFSNODE: 460, 0, 13, 27, 1525 > PIPE: 408, 0, 28, 35, 31594 > KNOTE: 68, 0, 0, 168, 5762 > socket: 356, 16434, 2327, 1699, 277319 > unpcb: 140, 65548, 42, 70, 12935 > ipq: 32, 2147, 0, 0, 2235 > udpcb: 180, 16434, 5, 127, 5531 > inpcb: 180, 16434, 2259, 1393, 258840 > tcpcb: 464, 16424, 2258, 1454, 258840 > tcptw: 48, 3354, 1, 389, 38232 > syncache: 100, 15366, 17, 334, 322283 > hostcache: 76, 15400, 246, 304, 6938 > tcpreass: 20, 4225, 1, 168, 4936 > sackhole: 20, 0, 8, 499, 23975962 > ripcb: 180, 16434, 0, 44, 11 > rtentry: 132, 0, 5, 53, 54 > pfsrctrpl: 100, 0, 0, 0, 0 > pfrulepl: 604, 0, 0, 0, 0 > pfstatepl: 260, 10005, 0, 0, 0 > pfaltqpl: 128, 0, 0, 0, 0 > pfpooladdrpl: 68, 0, 0, 0, 0 > pfrktable: 1240, 0, 0, 0, 0 > pfrkentry: 156, 0, 0, 0, 0 > pfrkentry2: 156, 0, 0, 0, 0 > pffrent: 16, 5075, 0, 0, 0 > pffrag: 48, 0, 0, 0, 0 > pffrcache: 48, 10062, 0, 0, 0 > pffrcent: 12, 50141, 0, 0, 0 > pfstatescrub: 28, 0, 0, 0, 0 > pfiaddrpl: 92, 0, 0, 0, 0 > pfospfen: 108, 0, 0, 0, 0 > pfosfp: 28, 0, 0, 0, 0 > IPFW dynamic rule zone: 108, 0, 0, 0, 0 > SWAPMETA: 276, 121576, 36, 20, 1613 > FFS inode: 132, 0, 27459, 1280, 4866112 > FFS1 dinode: 128, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 27459, 1206, 4866112 > > Intresting thing that everytime i use "vmstat 1" problem gone > and i save my uptime. > > Server is still under load and i see only one process in D state > > 85815 ?? D 0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0 > > Also i have found that it's much easy to trigger this state for httpd > when i use low (default) value for MaxSpareServers (I have about > 300 ~ 500 connections all the time) > > > -- > ========================================================================= > = Best regards, Nikolay Pavlov. <<<------------------------------------ = > ========================================================================= > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -- ========================================================================= = Best regards, Nikolay Pavlov. <<<------------------------------------ = =========================================================================