Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 22 May 2014 01:43:31 +1000 (EST)
From:      Bruce Evans <brde@optusnet.com.au>
To:        Gleb Smirnoff <glebius@freebsd.org>
Cc:        src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, Rui Paulo <rpaulo@felyko.com>, Bruce Evans <brde@optusnet.com.au>, Jack F Vogel <jfv@freebsd.org>, Julian Elischer <julian@freebsd.org>
Subject:   Re: svn commit: r266423 - in head/sys: conf dev/i40e modules/i40e
Message-ID:  <20140521190353.U974@besplex.bde.org>
In-Reply-To: <20140520171613.GM50679@FreeBSD.org>
References:  <201405190121.s4J1L3qA068339@svn.freebsd.org> <53796149.8060000@freebsd.org> <AF83F052-00D1-40E1-A427-58EDE0853D42@felyko.com> <20140520223516.R2836@besplex.bde.org> <20140520171613.GM50679@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 20 May 2014, Gleb Smirnoff wrote:

> On Tue, May 20, 2014 at 11:40:01PM +1000, Bruce Evans wrote:
> B> Also, verbose names break formatting.  E.g., netstat -r has 5 columns
> B> available under Netif for the driver name and device number.  netstat
> B> -i has about the same under Name (possibly 1 or 2 not directly under
> B> Name, but reserved for the Name column).  systat has 3 columns
> B> available, but with a more flexible format that truncates other info.
> B> All driver name+numbers are broken now on freefall:
>
> We must admit that nowadays 80x25 terminal is not enough :(

Yes, much smaller formats are needed now for small portable screens.
My phone is low-end so it is 29 or 30 wide.

80 is also too wide.  Most books use more like 60.  Most (?) newspapers
use more like 30, with many columns.

> Would be cool if most of tools (netstat, systat, etc...) could
> determine size of terminal and dynamically widen all their fields.

You mean, determine the size and dynamically narrow all their fields
:-).  This is hard to do.  Curses-based applications like top and
systat should do a combination of horizontal and vertical scrolling
and variant displays, but this is also hard to do.  top and systat
already have some variant displays but are too simple to do much.  You
can send raw output to a terminal or window that does the scrolling,
but this doesn't work for repeated output.

> Thus, tool can run w/o any abbreviations when run in a script mode,
> run abbreviated on a small terminal, and run verbose on a wide
> terminal.
> 
> This sounds like a generic library providing a special version
> of printf(3), which specifies minimal and maximum sizes for fields
> and when extra terminal width is available it distributes this
> width evenly between all fields. Name it 'elastic printf'.
> Sounds like a nice Google SoC project. Or might be that such
> library already exists.

Try doing that with systat -v output.  Even programming tools for
generating fixed forms like the ones used in systat are deficient
in FreeBSD.  All screen locations are almost hard-coded (just as
offsets) and hard to change.  Some of this is from primitive use
of curses, with no subwindows.

Before doing that, just handle arbitrary columnar output (with headers)
in a filter.  This can almost be done in an awk script.  Field
separators might be a problem.  Consider "vmstat 1 output".  It is
messed up by the header being 2 lines with the first line not being
in normal columnar format (some of its keywords are for multiple
columns).  A dumb filter can handle headers without knowing that they
are special provided they are in columnar format, but even this minor
complication seems to be difficult to handle without knowing what the
headers mean.

Here is a not so dumb filter using awk.  vmstat output is so broken in
-current that even this simple filter improves the formatting significantly.

@BEGIN {
@	columns = 79			# XXX
@}
@
@{
@	# Determine fields and field widths for current line.
@	# awk's field splitting feature turned out to be inadequate,
@	# and this would be even easier in C.
@	n = split($0, a, "")
@	f = 1
@	j = 1
@	while (j <= n) {
@		cpw[f] = 0;
@		cfw[f] = 0;
@		while (j <= n && a[j] == " ") {
@			cpw[f]++	# current (left) padding width
@			j++
@		}
@		while (j <= n && a[j] != " ") {
@			cfw[f]++	# current field width without padding
@			j++
@		}
@		fld[f] = substr($0, j - cfw[f], cfw[f])
@		f++
@	}
@	nf = f				# no need to use or trust NF
@
@	if (NR == 1 || NR < 10 && nf != anf) {
@		# Make current field widths active.  When NR > 1, this
@		# discards the previous active field widths.  Good enough
@		# for vmstat, where the widths from NR == 1 are garbage.
@		anf = nf
@		for (f = 1; f <= anf; f++)
@			afw[f] = cpw[f] + cfw[f]
@
@		# Convert current (unpadded) field widths to minumum (padded)
@		# field widths.
@		mfw[1] = cfw[1]
@		for (f = 2; f <= anf; f++)
@			mfw[f] = 1 + cfw[f]
@	} else if (nf != anf) {
@		# Some non-tabular line after warming up.  Probably an ornate
@		# line in the next header.  Too hard to handle properly.
@		printf("%.*s\n", columns, $0)
@		fflush(stdout)
@		next
@	} else {
@		# Update  and minimum field widths if this line needs
@		# wider fields.
@		if (afw[1] < cfw[1])
@			afw[1] = cfw[1]
@		len = afw[1]
@		if (mfw[1] < cfw[1])
@			mfw[1] = cfw[1]
@		for (f = 2; f <= nf; f++) {
@			if (afw[f] < cfw[f] + 1)
@				afw[f] = cfw[f] + 1
@			len += afw[f]
@			if (mfw[f] < cfw[f] + 1)
@				mfw[f] = cfw[f] + 1
@		}
@
@		# But if the line would be too wide, reset the active field
@		# widths to minimum ones.
@		if (len > columns)
@			for (f = 1; f <= nf; f++)
@				afw[f] = mfw[f]
@	}
@	s = ""
@	for (f = 1; f <= nf; f++)
@		s = s sprintf("%*s", afw[f], fld[f])
@	printf("%.*s\n", columns, s)
@	fflush(stdout)
@}

Examples:

vmstat wasn't broken in FreeBSD-5.2, at least in my version:

ttyv2:bde@besplex:/tmp/s1> vmstat 1
  procs      memory      page                    disks     faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr ad0 ad1   in   sy  cs us sy id
  0 0 0   48928 905436    8   0   0   0   6   0   0   0  235   52 596  0  0 100
  0 0 0   48928 905436    0   0   0   0   2   0   2   0  234  107 587  0  0 100
  0 0 0   48928 905436    0   0   0   0   0   0   0   0  231  107 580  0  0 100
...
  0 0 0   48928 905436    0   0   0   0   0   0   0   0  232  107 582  0  0 100
  procs      memory      page                    disks     faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr ad0 ad1   in   sy  cs us sy id
  0 0 0   48928 905436    0   0   0   0   0   0   0   0  231  120 581  0  0 100
  0 0 0   48928 905436    0   0   0   0   0   0   0   0  231  107 579  0  0 100
  0 0 0   48928 905436    0   0   0   0   0   0   0   0  231  115 580  0  0 100

Running this through the filter makes little difference:

ttyv2:bde@besplex:/tmp/s1> vmstat 1 |awk -f z
  procs      memory      page                    disks     faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr ad0 ad1   in   sy  cs us sy id
  0 1 0   49408 905232    8   0   0   0   6   0   0   0  235   52 596  0  0 100
  0 1 0   49480 905176   14   0   0   0   0   0   0   0  232  118 583  0  0 100
  0 1 0   49480 905176    0   0   0   0   0   0   0   0  231  117 582  0  0 100
...
  procs      memory      page                    disks     faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr ad0 ad1   in   sy  cs us sy  id
  0 1 0   49480 905176    0   0   0   0   0   0   0   0  233  115 587  0  0 100
  0 1 0   49480 905176    0   0   0   0   0   0   0   0  232  118 584  1  0  99
  1 1 0   49480 905176    0   0   0   0   0   0   0   0  231  109 581  0  0 100

The filter just eventually fixes the misformatting of 100% by right
justifying its header.  The formatting is %2.0f under "id", but 100 doesn't
fit.  100 also wouldn't fit under us or sy, but only id is often 100.  This
last worked in about 1985 where 100 was impossible for id too, since just
running vmstat every second took more than 1% CPU.

The broken output in -current looks like this in on freefall:

pts/18:bde@freefall:~/s> vmstat 1
  procs      memory      page                    disks     faults         cpu
  r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy i
d
  0 0 16   7312M   755M   751   2   2   0  1531 158   0   0 1850 1678  251  1  1
98
  0 0 16   7312M   755M    46   0   0   0    88  25   0   0  751  610 2456  0  0
100
  0 0 16   7384M   752M   551   0   0   0   186  25   0   0  722 1133 2374  0  0
100
...
  procs      memory      page                    disks     faults         cpu
  r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy i
d
  1 0 16   7541M   744M  2172   0   0   0  1566  28   0   0 1012 76327 18683  1
2 97
  1 0 16   7420M   746M   913   0   0   0  1744  27   0   0 1067 104617 20083  1
  4 95
  1 0 16   7487M   740M  1970   0   0   0   755  28  44  45 1496 124350 23887  1
  5 94

I used a vidcontrol screen dump to preserve its full uglyness.  Note that
the header formatting has been destroyed even though it is not dynamic:
- the first line of the header was not updated when the second line was
   expanded (except "cpu" was moved slightly, but it has an off by 1
   error in the move)
- the second header line is unconditionally too wide for an 80 column terminal

Note that avm and fre are now formatted in dehumanized (scientific)
form.  My 5.2 system doesn't have much memory, so it doesn't need this.
This makes avm and fre narrow enough to fit, but the formatting is otherwise
broken so they don't.  The "M" part never fits under the header.  This
might be intentional, but it looks strange.  It is actually clearly
unintentional since all the fields after avm and fre, through in, are
similarly misaligned with the header.

Then the bugs are from dynamic values being too large:
- despite expansion of the faults sy, the value is often too large to fit,
   so the fields from that point are shifted by 1 or 2 more more relative
   to the header.
- the cpu cs has been expanded too, but not enough for all cases
- the worst cases don't all occur on the same line, so the maximum shift
   relative to the header is 4.

When the output is not to a terminal, vmstat no longer dehumanizes some
large numbers:

pts/18:bde@freefall:~/s> vmstat 1|cat
  procs      memory      page                    disks     faults         cpu
  r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy i
d
  0 0 16 7504352  751956   752   2   2   0  1532 157   0   0 1850 1691  259  1  1
  98
  0 0 16 7500132  751900   118   0   0   0   164  50   1   1  726  476 2423  0  0
  100
...
  procs      memory      page                    disks     faults         cpu
  r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy i
d
  0 0 16 7500132  751924   547   0   0   0  1522  25   0   0  747 1218 2639  0  0
  100
  0 0 16 7500132  751924    46   0   0   0    88  25   0   0  668  348 2324  0  0
  100

Note that the dehumanization didn't even improve the formatting in this case.
The off-by-1 error in the field width and/or alignment gives just enough space
for the large avm.

freefall became less active while I was testing this, so many of the numbers
aren't so large now.

The filter cleans this up to:

pts/18:bde@freefall:~/s> vmstat 1|awk -f ../z
  procs      memory      page                    disks     faults         cpu
  r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy
2 0 16 7542196 812772 753  2  2  0 1532 157   0   0 1850 1694 260  1  1 98
0 0 16 7558808 809868 220  0  0  0 4117  27   0   0 6661 1392 18283  2  1 97
0 0 16 7558808 807060  46  0  0  0 3993  27  23  23 6640 1122 19186  2  1 97
1 0 16 7558808 804252  46  0  0  0 3995  27  28  27 6629 1131 19041  2  2 97
1 0 16 7750064 796536 1793  0  0  0 4761  27   0   0 6348 3978 17538  1  2 97
0 0 16 7750064 793648   46  0  0  0 4101  28   0   0 6304 1143 17556  1  1 97
...
  procs      memory      page                    disks     faults         cpu
r b  w     avm    fre  flt re pi po   fr  sr da0 da1   in   sy    cs us sy id
1 0 16 7591172 756388  112  0  0  0 5173  28   0   0 6389 1169 17547  2  1 97
0 0 16 7654924 752004 1967  0  0  0 6374  29   0   0 6621 3846 18279  2  2 97
0 0 16 7600364 754668  586  0  0  0 3416  28  54  54 3584 1618 12072  1  1 98
0 0 16 7409108 759696  104  0  0  0 1402  27   0   0  457  488  1428  0  0 100
0 0 16 7600364 754668 1804  0  0  0  934  25   0   0  454 3303  1544  0  0 100

It took a while to stabilize, and significant activity would push it to a
stable setting wanting more than 80 columns.  Dehumanized numbers for just
avm would help avoid this.

Note that the filter never shrinks the columns except once to squeeze out
spaces in the header.  It is too simple to do that.  It also responds too
quickly to transient changes.

netstat -i is also broken in -current (netstat -r in -current shows less
details so it fits easil):

pts/18:bde@freefall:~/s> netstat -i
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
  Coll
igb0   1500 <Link#1>      68:b5:99:b5:2a:02 2821384398     0     0 1778806163
   0     0
igb0      - 8.8.178.128/2 freefall          217503944     -     - 175357038
-     -
igb0      - fe80::6ab5:99 fe80::6ab5:99ff:f   104258     -     -   104297     -
     -
igb0      - freefall.free 2001:1900:2254:20 2576868711     -     - 1641398377
   -     -
igb0      - 8.8.178.141/3 people            24407600     -     -        0     -
     -
igb0      - people.freebs 2001:1900:2254:20  1171548     -     -   642454     -
     -
igb1   1500 <Link#2>      68:b5:99:b5:2a:03        0     0     0        0     0
     0
lo0   16384 <Link#3>                         2466053     0     0  2466037     0
     0
lo0       - localhost     ::1                  55872     -     -    55992     -
     -
lo0       - fe80::1%lo0   fe80::1%lo0              0     -     -        0     -
     -
lo0       - your-net      localhost          2402292     -     -  2410186     -
     -

The filter cleans this up to:

pts/18:bde@freefall:~/s> netstat -i|awk -f ../z
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
igb0 1500 <Link#1> 68:b5:99:b5:2a:02 2821393981     0     0 1778816441     0
igb0    - 8.8.178.128/2          freefall  217504375     -     -  175357732
igb0    - fe80::6ab5:99 fe80::6ab5:99ff:f     104261     -     -     104300
igb0    - freefall.free 2001:1900:2254:20 2576877564     -     - 1641408119
igb0    - 8.8.178.141/3            people   24407836     -     -          0
igb0    - people.freebs 2001:1900:2254:20    1171564     -     -     642460
igb1 1500      <Link#2> 68:b5:99:b5:2a:03          0     0     0          0
lo0   16384 <Link#3>                         2466061     0     0  2466045     0
lo0       - localhost     ::1                  55872     -     -    55992     -
lo0       - fe80::1%lo0   fe80::1%lo0              0     -     -        0     -
lo0       - your-net      localhost          2402300     -     -  2410194     -

It looks almost readable at first, but further examination shows that the
filter is not working as intended.  The Coll field got truncated away fairly
cleanly, and the first line for lo0 is unparseable since it has spaces
instead of "-" for the Address field.  The second of these breaks the header
hack.  After modifying the hack, it works to pass 2 copies through the filter
and discard the first half:

pts/18:bde@freefall:~/s> cat z
Name  Mtu       Network           Address      Ipkts Ierrs Idrop      Opkts Oer
igb0 1500      <Link#1> 68:b5:99:b5:2a:02 2822276782     0     0 1779538929
igb0    - 8.8.178.128/2          freefall  217533292     -     -  175425951
igb0    - fe80::6ab5:99 fe80::6ab5:99ff:f     104397     -     -     104436
igb0    - freefall.free 2001:1900:2254:20 2577700440     -     - 1642086286
igb0    - 8.8.178.141/3            people   24435004     -     -          0
igb0    - people.freebs 2001:1900:2254:20    1173149     -     -     643437
igb1 1500      <Link#2> 68:b5:99:b5:2a:03          0     0     0          0
lo0   16384 <Link#3>                         2467437     0     0  2467421     0
  lo0    -     localhost               ::1      55880     -     -      56000
  lo0    -   fe80::1%lo0       fe80::1%lo0          0     -     -          0
  lo0    -      your-net         localhost    2403657     -     -    2411562

Now Oerrs is truncated too (uncleanly), and the first lo0 line remains
unfixable.  It accidentally almost lines up with the header, since only
the Opkts field in the header got moved.

Note that the second igb0 field is obviously truncated before it gets to
the filter.  The full field can be seen using netstat -rn, and is 8 characters
wider.  Many of the other fields are truncated.  This is less clear, but
all of the Address fields that use the full field are probably truncated.
netstat -rn uses a separate format with very wide fields for the inet6 case
to avoid this problem.  This is ugly in a different way.

Bruce



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140521190353.U974>