Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 13 Dec 2005 12:16:08 -0500
From:      Mark Bucciarelli <mark@gaiahost.coop>
To:        freebsd-performance@freebsd.org
Subject:   [LONG] vmstat: What I/O is blocked and how to fix it?
Message-ID:  <20051213171608.GM2188@rabbit>

next in thread | raw e-mail | index | archive | help
This is a MIME-formatted message.  If you see this text it means that your
E-mail software does not support MIME-formatted messages.

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-2
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

[Originally sent to freebsd-questions, then I remembered this list (to 
which I am not subscribed, so please CC me on any replies.]

m

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-2
Content-Type: message/rfc822
Content-Disposition: inline

Return-path: <owner-freebsd-questions@freebsd.org>
Received: from solar.gaiahost.coop [64.95.78.5]
	by localhost with POP3 (fetchmail-6.2.5)
	for mark@localhost (single-drop); Tue, 13 Dec 2005 10:03:36 -0500 (EST)
Received: from mx2.freebsd.org (unverified [216.136.204.119]) by
	solar.gaiahost.coop (Rockliffe SMTPRA 4.5.6) with ESMTP id
	<B0006273351@solar.gaiahost.coop> for <mark@gaiahost.coop>; 
	Tue, 13 Dec 2005 09:55:47 -0500
Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18])
	by mx2.freebsd.org (Postfix) with ESMTP id 9D55C5B259;
	Tue, 13 Dec 2005 14:57:46 +0000 (GMT)
	(envelope-from owner-freebsd-questions@freebsd.org)
Received: from hub.freebsd.org (localhost [127.0.0.1])
	by hub.freebsd.org (Postfix) with ESMTP id 5924E16A422;
	Tue, 13 Dec 2005 14:57:46 +0000 (GMT)
	(envelope-from owner-freebsd-questions@freebsd.org)
X-Original-To: freebsd-questions@freebsd.org
Delivered-To: freebsd-questions@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 46B3B16A420
	for <freebsd-questions@freebsd.org>;
	Tue, 13 Dec 2005 14:54:20 +0000 (GMT)
	(envelope-from mark@gaiahost.coop)
Received: from biodiesel.gaiahost.coop (biodiesel.gaiahost.coop [64.95.78.120])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 8073C43D8C
	for <freebsd-questions@freebsd.org>;
	Tue, 13 Dec 2005 14:53:58 +0000 (GMT)
	(envelope-from mark@gaiahost.coop)
Received: from localhost (host-64-65-195-19.spr.choiceone.net
	[::ffff:64.65.195.19]) (AUTH: LOGIN mark@hubcapconsulting.com)
	by biodiesel.gaiahost.coop with esmtp; Tue, 13 Dec 2005 09:53:54 -0500
	id 006FC064.439EE083.00007756
Received: by localhost (sSMTP sendmail emulation);
	Tue, 13 Dec 2005 09:54:06 -0500
Date: Tue, 13 Dec 2005 09:54:06 -0500
From: Mark Bucciarelli <mark@gaiahost.coop>
To: freebsd-questions@freebsd.org
Message-ID: <20051213145406.GF2188@rabbit>
Mail-Followup-To: freebsd-questions@freebsd.org
Mime-Version: 1.0
Content-Type: multipart/mixed;
	boundary="=_biodiesel.gaiahost.coop-13953-1134494156-0001-3"
Content-Disposition: inline
User-Agent: Mutt/1.4.2.1i
Subject: [LONG] vmstat: What I/O is blocked and how to fix it?
X-BeenThere: freebsd-questions@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: User questions <freebsd-questions.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions>;
List-Post: <mailto:freebsd-questions@freebsd.org>
List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, 
	<mailto:freebsd-questions-request@freebsd.org?subject=subscribe>
Sender: owner-freebsd-questions@freebsd.org
Errors-To: owner-freebsd-questions@freebsd.org

This is a MIME-formatted message.  If you see this text it means that your
E-mail software does not support MIME-formatted messages.

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On two occasions recently, vmstat has showed me that a 
number of processes are blocked due to I/O.  At the same 
time, the number of disk transactions per second reported is 
a small fraction of the disk's capability.

I am thinking some kind of VM config will help, but based on what 
I have read, it is almost always best to let the default kernel 
behavior do its thing.  

Setup: 

    Dual PIII 1.0 GHz (256MB L1 and L2 cache)
    3GB RAM 
    custom kernel (SMP + PF)
    5.4-RELEASE
    one intel fxp0 network interface used (second unused)


Case 1: (da0 36GB SCSI)

    The first case is simple: running spamd-setup to load 
    the huge Composite Blacklist into spamd.  The only other 
    things running are sshd and spamd itself.

    When I run vmstat (vmstat -w 1 -c 10 output attached), 
    it shows ten processes blocked due to I/O, 1 runnable 
    and a total of four disk transactions during this 
    particular ten-second interval.  
    
    Accoring to postmark [1] da0 is capable of 250 
    transactions per second.

    Interrupts look fine (200 of the 300/sec is the clock, 
    right?).

    No swapping.  No network traffic to speak of.

    High CPU usage.

    Paging looks really high to me.  I have plenty of RAM 
    and swap--so why is it so high?  
    
    Is there a kernel param I can tweak to improve 
    performance? 

    Note:   While this was running, I tried running vmstat 
            -m and got the kmem_map too small error.  (I 
            read the FAQ entry and haven't yet rebuilt the 
            kernel using VM_KMEM_SIZE_MAX.)

    Should I make VM_KMEM_SIZE_MAX 400MB as mentioned in the FAQ?

    Any other kernel settings to switch from defaults when I 
    build the kernel?


Case 2: (mirror/gm0, da0 + da1 300GB SCSI)

    This case is more complex and more important.  This box 
    was heavily loaded, running SpamAssassin + ClamAv in one 
    jail, Courier-MTA + MySql in another, and dspamd + pf in 
    the jail host.  
    
    It was processing about one incoming email per second, 
    and had 20,000 entries in the spamd greylist and 4,000 
    in the whitelist.  The black list was not loaded.

    Courier-authlib was using MySQL to figure out where to 
    deliver the mail to for each of 20,000 virtual users.

    Courier-MTA was calling the ClamAV daemon via TCP/IP 
    from a courier filter and calling SpamAssassin via 
    TCP/IP from maildrop.

    Spamd (as in OpenBSD, not Apache) and courier were 
    hitting the disk hard--I watched top in "m" mode [2] and 
    over a five minute period these two apps had about 
    pretty much 50% each of the WRITES.

    The only thing that jumps out at me is that when things 
    finally do get written to disk, the context switches go 
    really high.

    Is this a VM issue of some sort?  The disk transactions 
    per second are really low; this 300GB SCSI can do 
    730/sec according to postmark (well, this was testing on 
    a single disk not with the RAID1 gmirror).
    

Thanks for any tips.

m

[1] /usr/ports/benchmarks/postmark 

[2] Where can I find docs on what the columns mean in top's 
    "m" mode?

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="vmstat.case1.out"

 procs      memory        page                      disks     faults   cpu
 r  b w     avm    fre    flt  re  pi  po    fr  sr da0 da1   in   sy  cs us sy id
 1 10 0  190424 1653924  1065   0   0   0  1337   0   0   0  796 2931 1971 3  8 89
 1 10 0  190520 1653824 17407   0   0   0 17401   0   0   0  356  111 280 44 56  0
 1 10 0  167376 1676992 23264   0   0   0 23264   0   0   0  359  119 301 40 23 37
 1 10 0  190808 1659240 27728   0   0   0 29152   0   0   0  351  114 297 39 14 47
 1 10 0  190936 1676864 18992   0   0   0 17544   0   4   0  352  115 286 39 45 17
 1 10 0  191032 1676808 17586   0   0   0 17586   0   0   0  349  115 291 41 52  7
 1 10 0  191160 1658572 28025   0   0   0 29374   0   0   0  347  114 270 42 10 47
 1 10 0  191320 1676668 24949   0   0   0 23576   0   0   0  346  122 280 36 16 47
 1 10 0  191384 1652968 17730   0   0   0 17724   0   0   0  350  111 278 41 59  0
 1 10 0  191512 1665912 20409   0   0   0 23688   0   0   0  373  113 304 39 40 21

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="vmstat.case2.out"

 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 0 27 0  345932 2622260  333   0   0   0  318   0   0   0  504  1261   842  8  3 89
 0 25 0  341644 2622880   78   0   0   0  215   0   0   0  535   195   575  0  5 95
 0 25 0  341644 2622880    0   0   0   0    0   0   0   0  455   159   458  0  6 94
 0 25 0  341644 2622880    5   0   0   0    5   0   5   5  488   165   572  0  4 96
 1 25 0  343856 2622408  270   0   0   0  126   0   3   4  472   867   706  1  7 92
 0 28 0  355672 2620588 2356   0   0   0 1701   0  44  44  749  6224  1988 24 13 63
 1 29 0  357616 2620356 1961   0   0   0 1757   0  44  44  819  7174  2333 46 15 39
 1 29 0  357620 2620360  809   0   0   0  735   0  25  25  700  2694  1247 65  8 27
 0 29 0  350148 2621460   79   0   0   0  349   0  18  17  654  1144   896 32  6 62
 1 28 0  348808 2621640  140   0   0   0  175   0   2   2  438   433   509  0  5 95
 1 29 0  351016 2621284  261   0   0   0  137   0  10  10  530   781   781  0  8 92
 0 29 0  351196 2620928 1790   0   0   0 1540   0  32  32  724  4914  1799 19 13 68
 7 27 0  366168 2618696 3657   0   0   0 2835   0  75  75 1003  9967  2644 58 21 21
 4 28 0  377836 2616740 2383   0   0   0 1719   0  44  43  898  7199  1926 84 16  0
 3 28 0  348996 2621212 2004   0   0   0 2851   0  74  75  948  7730  2200 62 15 23
 2 28 0  351156 2620636  396   0   0   0  335   0 119 127  928   951  1849  3 11 86
 1 29 0  358544 2619816 2006   0   0   0 1702   0 104  94 1046  7823  2761 33 17 49
 1 29 0  352904 2620540  791   0   0   0  920   0  21  22  582  2467  1138 23  8 69
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 0 30 0  353364 2620304 1004   0   0   0  837   0  16  15  652  3595  1202 33 10 57
 3 28 0  358624 2619544 2025   0   0   0 1693   0  42  42  925  5461  2103 20 13 67
 1 28 0  356484 2619684 1398   0   0   0 1396   0  28  28 1034 10604  4947 47 14 39
 3 27 0  362108 2618540 2623   0   0   0 2138   0 151 151 1156  6026  2356 67 24  9
 1 27 0  354620 2619880 1314   0   0   0 1615   0  39  40  808  5280  1401 83 12  4
 1 29 0  359088 2618976 2360   0   0   0 1960   0  47  46  888  7626  2158 41 18 41
 6 28 0  364524 2617952 2360   0   0   0 2031   0  37  37 1009 10575  3258 73 15 13
 2 30 0  357952 2618624 3010   0   0   0 2987   0  77  77 1123  9732  2629 64 19 17
 3 28 0  363976 2617728 2358   0   0   0 1962   0  43  43  987  7696  1977 73 13 14
 5 28 0  371768 2616336 2283   0   0   0 1833   0  37  37  862  6972  1760 80 17  4
 3 28 0  379872 2615124 3693   0   0   0 3158   0  77  78 1070 12627  3007 73 19  8
 8 29 0  374012 2616044  638   0   0   0  845   0  30  30  759  3607  1310 90 10  0
 2 31 0  363880 2617056 2532   0   0   0 2586   0  56  55 1100  8357  2252 74 17  9
 3 30 0  370596 2616180 1959   0   0   0 1702   0  43  43  839  8104  2062 76 14 11
 3 31 0  370756 2616252  173   0   0   0  152   0  12  12  596   606   854 96  4  0
 1 30 0  361572 2617508 2026   0   0   0 2190   0  44  45  838  7504  1644 63 13 25
 3 30 0  354344 2618260 1301   0   0   0 1433   0  30  29  716  4942  1371 50 10 41
 2 28 0  360372 2617440 2233   0   0   0 1871   0  33  33  745  6363  1656 42 13 45
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 4 29 0  369572 2615616 3118   0   0   0 2442   0  44  44 1104 11499  3007 61 18 20
 5 30 0  364748 2616200 2856   0   0   0 2805   0  78  77 1100  9827  2589 81 19  0
 2 29 0  373248 2614944 3685   0   0   0 3174   0  75  76 1095 13451  3296 70 26  4
 2 29 0  367108 2615596 1974   0   0   0 2005   0  62  63  883  6594  2055 77 21  2
 1 28 0  358020 2617080 1106   0   0   0 1392   0  34  33  667  6640  1427 57 38  4
 0 28 0  350552 2618204   68   0   0   0  339   0   9   9  441   983   511 21 25 54
 1 29 0  360044 2616796 1556   0   0   0 1104   0  17  17  612  3628  1252  5 12 82
 0 30 0  352804 2617552  199   0   0   0  551   0 375 375 2084  1935  6164 26 18 56
 1 28 0  358052 2616864 2037   0   0   0 1696   0 551 551 2999  6123 10286 47 32 20
 0 28 0  357948 2617000 2011   0   0   0 1931   0  41  42  754  6509  1661 39 13 48
 0 27 0  348212 2618536  237   0   0   0  578   0  13  13  498  1282   664 24  7 68
 1 27 0  355664 2617472 1489   0   0   0 1081   0  12  12  547  2983  1044 13 10 76
 2 28 0  350432 2618136  331   0   0   0  440   0  11  10  543  1486   846  7  5 88
 1 28 0  350424 2618096  896   0   0   0  831   0  38  38  765  2849  1568  9  9 81
 1 29 0  350640 2617780  524   0   0   0  418   0 170 182 1210  1469  2260 10 22 68
 4 27 0  365384 2616412 2592   0   0   0 2178   0  80  68  967  8095  2464 64 16 20
 0 27 0  322376 2633292  730   0   0   0 4882   0  34  34  611  3569  1248 65 10 25
 1 28 0  332068 2631740 1081   0   0   0  621   0  12  12  556  2683  1099  7  9 84
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 3 28 0  327040 2632400  570   0   0   0  750   0  25  26  868  3020  1662 15  9 76
 4 28 0  339792 2629996 2594   0   0   0 1861   0  36  36 1175  8680  2634 65 19 16
 2 29 0  341052 2629700 2682   0   0   0 2441   0  65  65 1130  8717  2452 76 16  8
 2 28 0  326736 2631800  130   0   0   0  627   0  17  16  641  2223  1003 14  7 80
 0 29 0  326804 2631596  311   0   0   0  243   0   7   7  659  1446  1135  3 10 88
 1 29 0  326828 2631324 1086   0   0   0  961   0  25  25  835  3648  1619 14  9 77
 1 29 0  334092 2629960 1032   0   0   0  830   0  17  18  904  4961  1147 63 12 25
 0 29 0  326776 2630852  813   0   0   0 1048   0  13  12 1037  3148  1203 28 13 58
 4 28 0  334316 2630088 2358   0   0   0 2581   0  58  58 1281 55999 20971 29 38 33
 2 27 0  344276 2628704 2424   0   0   0 1960   0  50  51  887  8730  2223 77 15  8
 6 26 0  335252 2630092 1008   0   0   0 1275   0  33  32  701  4373  1130 85 10  6
 1 26 0  329360 2631012 1615   0   0   0 1713   0  40  40  746  4744  1592 34 12 53
 0 26 0  321892 2632104   69   0   0   0  323   0  10  11  614  1045   833  6  7 88
 0 26 0  321892 2632104   15   0   0   0   15   0   6   6  499   265   648  0  6 94
 1 25 0  321812 2632172  326   0   0   0  330   0   8   8  549   761   746  1  4 94
 2 25 0  327236 2631288  927   0   0   0  635   0  15  15  516  2279   922 24  9 67
 0 25 0  319764 2632388   60   0   0   0  447   0 188 188 1111   864  2635  4 11 85
 1 25 0  319764 2632388    0   0   0   0    0   0   2   2  389   107   366  0  4 96
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 3 26 0  329384 2630900 1229   0   0   0  743   0  14  14  638  2543  1171  3 10 88
 0 26 0  321988 2631920  960   0   0   0 1156   0  30  30  691  3809  1321 53  9 38
 1 26 0  322024 2631852   44   0   0   0   29   0  10  10  591   450   901  0  7 93
 2 27 0  338424 2629412 1875   0   0   0 1170   0  27  27  678  5395  1730 10 12 78
 1 27 0  331624 2630436  346   0   0   0  547   0   9   8  462  1473   693 61  6 33
 0 28 0  326352 2631132  698   0   0   0  789   0  13  13  618  2052  1007  9  6 85
 1 28 0  326476 2630648   75   0   0   0   17   0   3   3  765  1518  1191  4  6 90
 1 30 0  338916 2628664 1850   0   0   0 1556   0 100 107 1458 28808 12099 18 36 45
 0 30 0  330780 2629956  379   0   0   0  732   0  84  78  837  2851  1877 13 11 76
 0 29 0  328640 2630260  172   0   0   0  241   0   9   9  512   850  1012  0  8 92
 0 29 0  328640 2630260    0   0   0   0    4   0   2   2  389   121   369  0  4 96
 1 30 0  331940 2629672  884   0   0   0  681   0  13  12  607  2761  1047 17  7 76
 0 29 0  328640 2630216   40   0   0   0  168   0   9   9  508   425   641  3  6 91
 0 29 0  328640 2630212   11   0   0   0   10   0   4   4  548   230   644  0  5 95
 0 29 0  328640 2630212    5   0   0   0    5   0   2   2  430   219   470  0  4 96
 0 29 0  328640 2630208    6   0   0   0    5   0   2   2  437   256   481  1  4 96
 0 28 0  326496 2630516  277   0   0   0  348   0   8   8  526   780   788  0  6 94
 0 29 0  328728 2630052  274   0   0   0  126   0   2   2  451   802   604  1  7 92
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 0 29 0  328636 2630204  842   0   0   0  807   0  27  27  619  2368  1140 12  8 80
 0 29 0  328636 2630200    0   0   0   0    0   0   0   0  492   113   457  0  4 96
 0 30 0  330732 2629900  333   0   0   0  233   0   2   2  503   687   607  0  6 94
 1 30 0  330748 2629864   15   0   0   0    1   0   1   1  400   189   401  0  6 94
 2 29 0  338296 2628512 1340   0   0   0  911   0  21  21  910  4186  1712 23 11 66
 1 31 0  338024 2626652 1223   0   0   0 1364   0  51  51 2972 40356 18751 38 32 30
 0 31 0  333028 2628012  318   0   0   0 1159   0  32  33  757 67408 35201  7 43 50
 0 31 0  338336 2627192 1044   0   0   0  909   0  86  98  923  3573  2295 24 11 65
 0 31 0  333088 2627968 1367   0   0   0 1407   0  65  53  734  4235  1442 20  9 71
 0 30 0  330876 2628372   29   0   0   0  130   0   0   0  405   191   401  0  5 95
 0 29 0  328680 2628740  152   0   0   0  246   0   2   2  428   459   495  1  6 93
 2 29 0  330924 2628308  476   0   0   0  274   0 552 552 2776  1862 14249  3 24 72
 0 30 0  337484 2627268 1119   0   0   0  877   0  18  18  875 11192  5634 16 15 69
 3 30 0  345588 2625052 1285   0   0   0  675   0  13  13  748  2201  1059 53  7 40
 1 29 0  336604 2626380  344   0   0   0  653   0  23  23  557  1605   911 54  8 38
 0 29 0  329528 2627284  159   0   0   0  407   0  15  15  581  1541   679 12  5 83
 0 29 0  329528 2627284    5   0   0   0    5   0   2   2  397   159   391  0  4 96
 0 29 0  329528 2627300    5   0   0   0    9   0   3   3  474   177   539  0  4 96
 procs      memory       page                     disks     faults         cpu
 r  b w     avm     fre  flt  re  pi  po   fr  sr da0 da1   in    sy    cs us sy id
 2 28 0  329548 2627280    5   0   0   0    5   0  12  12  539   246   678  0  5 94
 0 30 0  331624 2627356  232   0   0   0  219   0  56  56  737   552   968  2 17 82
 2 30 0  331724 2627304  300   0   0   0  246   0   2   2  402   676   518  1  6 93
 1 29 0  337012 2626560 1654   0   0   0 1361   0  35  35  695  4538  1521 25 11 64
 0 28 0  327388 2627956   96   0   0   0  429   0  12  12  464  1038   605 14  5 81
 0 28 0  327388 2627956    5   0   0   0    5   0   2   2  473   191   526  0  4 96
 2 28 0  329632 2627500  280   0   0   0  130   0   4   4  518   739   751  0  5 95
 0 28 0  327388 2627916  877   0   0   0  934   0  24  24  695  3220  1214 21 10 70
 0 28 0  327388 2627920  110   0   0   0  113   0   0   0  411   324   406  0  6 94
 1 27 0  327388 2627920    0   0   0   0    4   0   2   2  404   128   387  0  4 96
 2 26 0  325268 2628176  480   0   0   0  470   0   2   2  460   958   649  1  6 93
 3 26 0  332712 2627008  934   0   0   0  641   0  19  19  831 10859  4034 16 13 71

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
--=_biodiesel.gaiahost.coop-13953-1134494156-0001-3--

--=_biodiesel.gaiahost.coop-13953-1134494156-0001-2--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051213171608.GM2188>