From owner-svn-src-all@FreeBSD.ORG Wed May 21 16:06:14 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C74A66BF; Wed, 21 May 2014 16:06:14 +0000 (UTC) Received: from mail110.syd.optusnet.com.au (mail110.syd.optusnet.com.au [211.29.132.97]) by mx1.freebsd.org (Postfix) with ESMTP id 4867724CB; Wed, 21 May 2014 16:06:14 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail110.syd.optusnet.com.au (Postfix) with ESMTPS id 68BE0781770; Thu, 22 May 2014 01:43:31 +1000 (EST) Date: Thu, 22 May 2014 01:43:31 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Gleb Smirnoff Subject: Re: svn commit: r266423 - in head/sys: conf dev/i40e modules/i40e In-Reply-To: <20140520171613.GM50679@FreeBSD.org> Message-ID: <20140521190353.U974@besplex.bde.org> References: <201405190121.s4J1L3qA068339@svn.freebsd.org> <53796149.8060000@freebsd.org> <20140520223516.R2836@besplex.bde.org> <20140520171613.GM50679@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=QIpRGG7L c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=JD0ISiim65UA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=AXO4mayRf3EEWKXxS8gA:9 a=ItVfWD_4TXz5l3mK:21 a=ur02btxSU7qdEW1z:21 a=CjuIK1q_8ugA:10 a=xW-gOTbM36EA:10 a=y9Lmiy5wWV0A:10 a=HwfidlGBIzgA:10 a=O_wSirOHFWgA:10 Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, Rui Paulo , Bruce Evans , Jack F Vogel , Julian Elischer X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 16:06:14 -0000 On Tue, 20 May 2014, Gleb Smirnoff wrote: > On Tue, May 20, 2014 at 11:40:01PM +1000, Bruce Evans wrote: > B> Also, verbose names break formatting. E.g., netstat -r has 5 columns > B> available under Netif for the driver name and device number. netstat > B> -i has about the same under Name (possibly 1 or 2 not directly under > B> Name, but reserved for the Name column). systat has 3 columns > B> available, but with a more flexible format that truncates other info. > B> All driver name+numbers are broken now on freefall: > > We must admit that nowadays 80x25 terminal is not enough :( Yes, much smaller formats are needed now for small portable screens. My phone is low-end so it is 29 or 30 wide. 80 is also too wide. Most books use more like 60. Most (?) newspapers use more like 30, with many columns. > Would be cool if most of tools (netstat, systat, etc...) could > determine size of terminal and dynamically widen all their fields. You mean, determine the size and dynamically narrow all their fields :-). This is hard to do. Curses-based applications like top and systat should do a combination of horizontal and vertical scrolling and variant displays, but this is also hard to do. top and systat already have some variant displays but are too simple to do much. You can send raw output to a terminal or window that does the scrolling, but this doesn't work for repeated output. > Thus, tool can run w/o any abbreviations when run in a script mode, > run abbreviated on a small terminal, and run verbose on a wide > terminal. > > This sounds like a generic library providing a special version > of printf(3), which specifies minimal and maximum sizes for fields > and when extra terminal width is available it distributes this > width evenly between all fields. Name it 'elastic printf'. > Sounds like a nice Google SoC project. Or might be that such > library already exists. Try doing that with systat -v output. Even programming tools for generating fixed forms like the ones used in systat are deficient in FreeBSD. All screen locations are almost hard-coded (just as offsets) and hard to change. Some of this is from primitive use of curses, with no subwindows. Before doing that, just handle arbitrary columnar output (with headers) in a filter. This can almost be done in an awk script. Field separators might be a problem. Consider "vmstat 1 output". It is messed up by the header being 2 lines with the first line not being in normal columnar format (some of its keywords are for multiple columns). A dumb filter can handle headers without knowing that they are special provided they are in columnar format, but even this minor complication seems to be difficult to handle without knowing what the headers mean. Here is a not so dumb filter using awk. vmstat output is so broken in -current that even this simple filter improves the formatting significantly. @BEGIN { @ columns = 79 # XXX @} @ @{ @ # Determine fields and field widths for current line. @ # awk's field splitting feature turned out to be inadequate, @ # and this would be even easier in C. @ n = split($0, a, "") @ f = 1 @ j = 1 @ while (j <= n) { @ cpw[f] = 0; @ cfw[f] = 0; @ while (j <= n && a[j] == " ") { @ cpw[f]++ # current (left) padding width @ j++ @ } @ while (j <= n && a[j] != " ") { @ cfw[f]++ # current field width without padding @ j++ @ } @ fld[f] = substr($0, j - cfw[f], cfw[f]) @ f++ @ } @ nf = f # no need to use or trust NF @ @ if (NR == 1 || NR < 10 && nf != anf) { @ # Make current field widths active. When NR > 1, this @ # discards the previous active field widths. Good enough @ # for vmstat, where the widths from NR == 1 are garbage. @ anf = nf @ for (f = 1; f <= anf; f++) @ afw[f] = cpw[f] + cfw[f] @ @ # Convert current (unpadded) field widths to minumum (padded) @ # field widths. @ mfw[1] = cfw[1] @ for (f = 2; f <= anf; f++) @ mfw[f] = 1 + cfw[f] @ } else if (nf != anf) { @ # Some non-tabular line after warming up. Probably an ornate @ # line in the next header. Too hard to handle properly. @ printf("%.*s\n", columns, $0) @ fflush(stdout) @ next @ } else { @ # Update and minimum field widths if this line needs @ # wider fields. @ if (afw[1] < cfw[1]) @ afw[1] = cfw[1] @ len = afw[1] @ if (mfw[1] < cfw[1]) @ mfw[1] = cfw[1] @ for (f = 2; f <= nf; f++) { @ if (afw[f] < cfw[f] + 1) @ afw[f] = cfw[f] + 1 @ len += afw[f] @ if (mfw[f] < cfw[f] + 1) @ mfw[f] = cfw[f] + 1 @ } @ @ # But if the line would be too wide, reset the active field @ # widths to minimum ones. @ if (len > columns) @ for (f = 1; f <= nf; f++) @ afw[f] = mfw[f] @ } @ s = "" @ for (f = 1; f <= nf; f++) @ s = s sprintf("%*s", afw[f], fld[f]) @ printf("%.*s\n", columns, s) @ fflush(stdout) @} Examples: vmstat wasn't broken in FreeBSD-5.2, at least in my version: ttyv2:bde@besplex:/tmp/s1> vmstat 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 0 0 48928 905436 8 0 0 0 6 0 0 0 235 52 596 0 0 100 0 0 0 48928 905436 0 0 0 0 2 0 2 0 234 107 587 0 0 100 0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 107 580 0 0 100 ... 0 0 0 48928 905436 0 0 0 0 0 0 0 0 232 107 582 0 0 100 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 120 581 0 0 100 0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 107 579 0 0 100 0 0 0 48928 905436 0 0 0 0 0 0 0 0 231 115 580 0 0 100 Running this through the filter makes little difference: ttyv2:bde@besplex:/tmp/s1> vmstat 1 |awk -f z procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 1 0 49408 905232 8 0 0 0 6 0 0 0 235 52 596 0 0 100 0 1 0 49480 905176 14 0 0 0 0 0 0 0 232 118 583 0 0 100 0 1 0 49480 905176 0 0 0 0 0 0 0 0 231 117 582 0 0 100 ... procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad1 in sy cs us sy id 0 1 0 49480 905176 0 0 0 0 0 0 0 0 233 115 587 0 0 100 0 1 0 49480 905176 0 0 0 0 0 0 0 0 232 118 584 1 0 99 1 1 0 49480 905176 0 0 0 0 0 0 0 0 231 109 581 0 0 100 The filter just eventually fixes the misformatting of 100% by right justifying its header. The formatting is %2.0f under "id", but 100 doesn't fit. 100 also wouldn't fit under us or sy, but only id is often 100. This last worked in about 1985 where 100 was impossible for id too, since just running vmstat every second took more than 1% CPU. The broken output in -current looks like this in on freefall: pts/18:bde@freefall:~/s> vmstat 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i d 0 0 16 7312M 755M 751 2 2 0 1531 158 0 0 1850 1678 251 1 1 98 0 0 16 7312M 755M 46 0 0 0 88 25 0 0 751 610 2456 0 0 100 0 0 16 7384M 752M 551 0 0 0 186 25 0 0 722 1133 2374 0 0 100 ... procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i d 1 0 16 7541M 744M 2172 0 0 0 1566 28 0 0 1012 76327 18683 1 2 97 1 0 16 7420M 746M 913 0 0 0 1744 27 0 0 1067 104617 20083 1 4 95 1 0 16 7487M 740M 1970 0 0 0 755 28 44 45 1496 124350 23887 1 5 94 I used a vidcontrol screen dump to preserve its full uglyness. Note that the header formatting has been destroyed even though it is not dynamic: - the first line of the header was not updated when the second line was expanded (except "cpu" was moved slightly, but it has an off by 1 error in the move) - the second header line is unconditionally too wide for an 80 column terminal Note that avm and fre are now formatted in dehumanized (scientific) form. My 5.2 system doesn't have much memory, so it doesn't need this. This makes avm and fre narrow enough to fit, but the formatting is otherwise broken so they don't. The "M" part never fits under the header. This might be intentional, but it looks strange. It is actually clearly unintentional since all the fields after avm and fre, through in, are similarly misaligned with the header. Then the bugs are from dynamic values being too large: - despite expansion of the faults sy, the value is often too large to fit, so the fields from that point are shifted by 1 or 2 more more relative to the header. - the cpu cs has been expanded too, but not enough for all cases - the worst cases don't all occur on the same line, so the maximum shift relative to the header is 4. When the output is not to a terminal, vmstat no longer dehumanizes some large numbers: pts/18:bde@freefall:~/s> vmstat 1|cat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i d 0 0 16 7504352 751956 752 2 2 0 1532 157 0 0 1850 1691 259 1 1 98 0 0 16 7500132 751900 118 0 0 0 164 50 1 1 726 476 2423 0 0 100 ... procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy i d 0 0 16 7500132 751924 547 0 0 0 1522 25 0 0 747 1218 2639 0 0 100 0 0 16 7500132 751924 46 0 0 0 88 25 0 0 668 348 2324 0 0 100 Note that the dehumanization didn't even improve the formatting in this case. The off-by-1 error in the field width and/or alignment gives just enough space for the large avm. freefall became less active while I was testing this, so many of the numbers aren't so large now. The filter cleans this up to: pts/18:bde@freefall:~/s> vmstat 1|awk -f ../z procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy 2 0 16 7542196 812772 753 2 2 0 1532 157 0 0 1850 1694 260 1 1 98 0 0 16 7558808 809868 220 0 0 0 4117 27 0 0 6661 1392 18283 2 1 97 0 0 16 7558808 807060 46 0 0 0 3993 27 23 23 6640 1122 19186 2 1 97 1 0 16 7558808 804252 46 0 0 0 3995 27 28 27 6629 1131 19041 2 2 97 1 0 16 7750064 796536 1793 0 0 0 4761 27 0 0 6348 3978 17538 1 2 97 0 0 16 7750064 793648 46 0 0 0 4101 28 0 0 6304 1143 17556 1 1 97 ... procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 1 0 16 7591172 756388 112 0 0 0 5173 28 0 0 6389 1169 17547 2 1 97 0 0 16 7654924 752004 1967 0 0 0 6374 29 0 0 6621 3846 18279 2 2 97 0 0 16 7600364 754668 586 0 0 0 3416 28 54 54 3584 1618 12072 1 1 98 0 0 16 7409108 759696 104 0 0 0 1402 27 0 0 457 488 1428 0 0 100 0 0 16 7600364 754668 1804 0 0 0 934 25 0 0 454 3303 1544 0 0 100 It took a while to stabilize, and significant activity would push it to a stable setting wanting more than 80 columns. Dehumanized numbers for just avm would help avoid this. Note that the filter never shrinks the columns except once to squeeze out spaces in the header. It is too simple to do that. It also responds too quickly to transient changes. netstat -i is also broken in -current (netstat -r in -current shows less details so it fits easil): pts/18:bde@freefall:~/s> netstat -i Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll igb0 1500 68:b5:99:b5:2a:02 2821384398 0 0 1778806163 0 0 igb0 - 8.8.178.128/2 freefall 217503944 - - 175357038 - - igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104258 - - 104297 - - igb0 - freefall.free 2001:1900:2254:20 2576868711 - - 1641398377 - - igb0 - 8.8.178.141/3 people 24407600 - - 0 - - igb0 - people.freebs 2001:1900:2254:20 1171548 - - 642454 - - igb1 1500 68:b5:99:b5:2a:03 0 0 0 0 0 0 lo0 16384 2466053 0 0 2466037 0 0 lo0 - localhost ::1 55872 - - 55992 - - lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0 - - lo0 - your-net localhost 2402292 - - 2410186 - - The filter cleans this up to: pts/18:bde@freefall:~/s> netstat -i|awk -f ../z Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs igb0 1500 68:b5:99:b5:2a:02 2821393981 0 0 1778816441 0 igb0 - 8.8.178.128/2 freefall 217504375 - - 175357732 igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104261 - - 104300 igb0 - freefall.free 2001:1900:2254:20 2576877564 - - 1641408119 igb0 - 8.8.178.141/3 people 24407836 - - 0 igb0 - people.freebs 2001:1900:2254:20 1171564 - - 642460 igb1 1500 68:b5:99:b5:2a:03 0 0 0 0 lo0 16384 2466061 0 0 2466045 0 lo0 - localhost ::1 55872 - - 55992 - lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0 - lo0 - your-net localhost 2402300 - - 2410194 - It looks almost readable at first, but further examination shows that the filter is not working as intended. The Coll field got truncated away fairly cleanly, and the first line for lo0 is unparseable since it has spaces instead of "-" for the Address field. The second of these breaks the header hack. After modifying the hack, it works to pass 2 copies through the filter and discard the first half: pts/18:bde@freefall:~/s> cat z Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oer igb0 1500 68:b5:99:b5:2a:02 2822276782 0 0 1779538929 igb0 - 8.8.178.128/2 freefall 217533292 - - 175425951 igb0 - fe80::6ab5:99 fe80::6ab5:99ff:f 104397 - - 104436 igb0 - freefall.free 2001:1900:2254:20 2577700440 - - 1642086286 igb0 - 8.8.178.141/3 people 24435004 - - 0 igb0 - people.freebs 2001:1900:2254:20 1173149 - - 643437 igb1 1500 68:b5:99:b5:2a:03 0 0 0 0 lo0 16384 2467437 0 0 2467421 0 lo0 - localhost ::1 55880 - - 56000 lo0 - fe80::1%lo0 fe80::1%lo0 0 - - 0 lo0 - your-net localhost 2403657 - - 2411562 Now Oerrs is truncated too (uncleanly), and the first lo0 line remains unfixable. It accidentally almost lines up with the header, since only the Opkts field in the header got moved. Note that the second igb0 field is obviously truncated before it gets to the filter. The full field can be seen using netstat -rn, and is 8 characters wider. Many of the other fields are truncated. This is less clear, but all of the Address fields that use the full field are probably truncated. netstat -rn uses a separate format with very wide fields for the inet6 case to avoid this problem. This is ugly in a different way. Bruce