Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 10 Mar 2017 15:39:08 -0800
From:      "Tony Hain" <tony@tndh.net>
To:        <freebsd-arm@freebsd.org>
Cc:        "'Ian Lepore'" <ian@freebsd.org>
Subject:   RE: BBB asymmetric stack?
Message-ID:  <066b01d299f7$8336bc60$89a43520$@tndh.net>

next in thread | raw e-mail | index | archive | help
Follow up:
Using OWAMP (http://software.internet2.edu/owamp/) there is a clear
asymmetry in the FreeBSD-12 arm stack. Measurements originating from the BBB
consistently take longer than those originating in the opposite direction,
and jitter is significantly worse when FreeBSD-12 arm is doing the echo.
This level of error results in an offset for ntp.
FreeBSD12_BBB -> ARM Cortex-A8 1.0GHz
Rpi_raspberrypi_4.1.15+ -> model B ARM Cortex-A53 1.2 GHz
FreeBSD8_i386a -> hw.model: Intel(R) Celeron(R) CPU 2.40GHz
FreeBSD8_i386b -> hw.model: Intel(R) Pentium(R) 4 CPU 1.80GHz

BBB & Rpi connected to the same switch. i386a & i386b are 2 local router
hops off that switch with routing symmetry verified.  Measurements done
sequentially to avoid any cross impact :::::

FreeBSD12_BBB ->
# owping -c 1000 -n u 2001:470:e930:1240:-i386a-
Approximately 103.2 seconds until results available

--- owping statistics from [2001:470:e930:2240:-BBB-]:8769 to
[2001:470:e930:1240:-i386a-]:2259 ---
SID:    fe044c0adc6da10e3acf2a8a67086b90
first:  2017-03-10T14:02:55.463
last:   2017-03-10T14:04:38.162
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 288/500/1.74e+04 us, (err=3.02 us)
one-way jitter = 100 us (P95-P50)
Hops = 2 (consistently)
no reordering

--- owping statistics from [2001:470:e930:1240:-i386a-]:2263 to
[2001:470:e930:2240:-BBB-]:8883 ---
SID:    fefbf6c6dc6da10e5380d38f54bbd8b6
first:  2017-03-10T14:02:55.466
last:   2017-03-10T14:04:34.525
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 350/500/1.99e+04 us, (err=3.02 us)
one-way jitter = 300 us (P95-P50)
Hops = 2 (consistently)
no reordering


FreeBSD8_i386a ->
# owping -c 1000 -n u 2001:470:e930:2240:-BBB-
Approximately 103.3 seconds until results available

--- owping statistics from [2001:470:e930:1240:-i386a-]:10791 to
[2001:470:e930:2240:-BBB-]:4212 ---
SID:    fefbf6c6dc6da19d78d6292c35ef1187
first:  2017-03-10T14:05:18.697
last:   2017-03-10T14:06:54.748
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 345/500/2.18e+04 us, (err=3.02 us)
one-way jitter = 400 us (P95-P50)
Hops = 2 (consistently)
no reordering

--- owping statistics from [2001:470:e930:2240:-BBB-]:4265 to
[2001:470:e930:1240:-i386a-]:42582 ---
SID:    fe044c0adc6da19d7cfebce7630aea48
first:  2017-03-10T14:05:18.797
last:   2017-03-10T14:06:56.113
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 295/500/6.32e+04 us, (err=3.02 us)
one-way jitter = 100 us (P95-P50)
Hops = 2 (consistently)
no reordering


FreeBSD12_BBB ->
# owping -c 1000 -n u 2001:470:e930:12c0:-i386b-
Approximately 103.3 seconds until results available

--- owping statistics from [2001:470:e930:2240:-BBB-]:8855 to
[2001:470:e930:12c0:-i386b-]:2318 ---
SID:    fe794e70dc6da2372cea6331afc13440
first:  2017-03-10T14:07:52.407
last:   2017-03-10T14:09:29.933
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 254/400/1.42e+04 us, (err=3.02 us)
one-way jitter = 100 us (P95-P50)
Hops = 2 (consistently)
no reordering

--- owping statistics from [2001:470:e930:12c0:-i386b-]:2300 to
[2001:470:e930:2240:-BBB-]:8887 ---
SID:    fefbf6c6dc6da237329d336cf9a12126
first:  2017-03-10T14:07:52.463
last:   2017-03-10T14:09:30.625
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 322/500/1.36e+04 us, (err=3.02 us)
one-way jitter = 200 us (P95-P50)
Hops = 2 (consistently)
no reordering


FreeBSD8_i386b ->
# owping -c 1000 -n u 2001:470:e930:2240:-BBB-
Approximately 103.2 seconds until results available

--- owping statistics from [2001:470:e930:12c0:-i386b-]:57055 to
[2001:470:e930:2240:-BBB-]:4211 ---
SID:    fefbf6c6dc6da2a71e6d9f99dfe5be1b
first:  2017-03-10T14:09:44.324
last:   2017-03-10T14:11:26.464
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 305/500/2.57e+04 us, (err=4.02 us)
one-way jitter = 200 us (P95-P50)
Hops = 2 (consistently)
no reordering

--- owping statistics from [2001:470:e930:2240:-BBB-]:4231 to
[2001:470:e930:12c0:-i386b-]:64099 ---
SID:    fe794e70dc6da2a7243e1ee38651110b
first:  2017-03-10T14:09:44.355
last:   2017-03-10T14:11:25.457
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 254/400/2.5e+04 us, (err=4.02 us)
one-way jitter = 100 us (P95-P50)
Hops = 2 (consistently)
no reordering


FreeBSD12_BBB ->
# owping -c 1000 -n u 2001:470:e930:2240:-Rpi-
Approximately 103.4 seconds until results available

--- owping statistics from [2001:470:e930:2240:-BBB-]:8928 to
[2001:470:e930:2240:-Rpi-]:4226 ---
SID:    4b031abcdc6da34676422beb5014dbd9
first:  2017-03-10T14:12:23.808
last:   2017-03-10T14:14:03.739
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 285/400/1.82e+04 us, (err=2.01 us)
one-way jitter = 200 us (P95-P50)
TTL not reported
no reordering

--- owping statistics from [2001:470:e930:2240:-Rpi-]:4236 to
[2001:470:e930:2240:-BBB-]:8813 ---
SID:    fefbf6c6dc6da346861b906566678c71
first:  2017-03-10T14:12:23.828
last:   2017-03-10T14:14:12.996
1000 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 243/400/2.34e+04 us, (err=2.01 us)
one-way jitter = 300 us (P95-P50)
TTL not reported
no reordering


raspberrypi_4.1.15+ ->
# owping -c 100 -n u 2001:470:e930:2240:-BBB-
Approximately 12.9 seconds until results available

--- owping statistics from [2001:470:e930:2240:-Rpi-]:8906 to
[2001:470:e930:2240:-BBB-]:4214 ---
SID:    fefbf6c6dc6da414e9617290304594ab
first:  2017-03-10T14:15:49.962
last:   2017-03-10T14:16:00.106
100 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 255/400/2.71e+03 us, (err=2.01 us)
one-way jitter = 300 us (P95-P50)
TTL not reported
no reordering

--- owping statistics from [2001:470:e930:2240:-BBB-]:4210 to
[2001:470:e930:2240:-Rpi-]:8912 ---
SID:    4b031abcdc6da414f059a52f753769e5
first:  2017-03-10T14:15:50.032
last:   2017-03-10T14:16:01.078
100 sent, 0 lost (0.000%), 0 duplicates
one-way delay min/median/max = 289/400/707 us, (err=2.01 us)
one-way jitter = 200 us (P95-P50)
TTL not reported
no reordering



> -----Original Message-----
> From: Tony Hain [mailto:tony@tndh.net]
> Sent: Sunday, March 05, 2017 6:04 PM
> To: 'freebsd-arm@freebsd.org'
> Cc: 'Ian Lepore'
> Subject: BBB asymmetric stack?
> 
> FreeBSD 12.0-CURRENT #0 r314118M: Wed Feb 22 22:49:19 PST 2017
> 
> Trying to calibrate the ntp feed on the BBB I was suspecting some problem
> with the dmtpps driver, but then realized that network asymmetry would be
> more likely to explain the persistent offset. It would appear that there
is an
> asymmetry within the stack on the am335x based system.
> 
> ping6 Rpi -> BBB
> 10 packets transmitted, 10 received, 0% packet loss, time 9006ms rtt
> min/avg/max/mdev = 0.503/0.567/0.642/0.047 ms
> 
> ping6 BBB -> Rpi
> 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip
> min/avg/max/std-dev = 0.878/0.967/1.141/0.134 ms
> 
> Those are on adjacent ports on the same 100mbps switch.
> 
> ---
> 
> ping6 Fbsd-8 -> BBB
> 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip
> min/avg/max/std-dev = 0.599/0.734/1.041/0.112 ms
> 
> ping6 BBB -> Fbsd-8
> 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip
> min/avg/max/std-dev = 0.855/0.970/1.816/0.296 ms
> 
> Those are one router hop apart.
> 
> ---
> 
> Even localhost appears to indicate a problem in the FreeBSD-arm side of
this:
> pi# ping6 ::1
> PING ::1(::1) 56 data bytes
> 64 bytes from ::1: icmp_seq=1 ttl=64 time=0.220 ms
> 64 bytes from ::1: icmp_seq=2 ttl=64 time=0.222 ms
> 64 bytes from ::1: icmp_seq=3 ttl=64 time=0.221 ms
> 
> i386# ping6 ::1
> PING6(56=40+8+8 bytes) ::1 --> ::1
> 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.185 ms
> 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.135 ms
> 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.129 ms
> 
> bbb# ping6 ::1
> PING6(56=40+8+8 bytes) ::1 --> ::1
> 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.672 ms
> 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.477 ms
> 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.464 ms
> 
> 
> I realize the loopback ping by itself is not a good indicator of an
asymmetry
> problem, but FreeBSD 8 on i386 systems ping::1 at ~ 130us, while all the
> FreeBSD 10/11 VMs (Hyper-V & VMware) and the pi ping ::1 at ~220us. The
> issue is that the time is always longer when the BBB initiates the ping,
which
> would imply some transmit delay that does not exist when responding to an
> incoming ping, but it could be simply getting the reply back to user space
> which only happens in the BBB initiate case.
> 
> ====
> The ntp perspective might shed some light on which direction.
> 
> pi
> pi# ntpq -p
>      remote           refid      st t when poll reach   delay   offset
jitter
> ==========================================================
> ====================
>  GPS_NMEA(0)     .GPS.            0 s 342d   16    0    0.000  -29.245
0.000
> oPPS(0)          .PPS.            0 s    6   16  377    0.000    0.000
0.002
> *172.20.0.18     .PPS.            1 u   12   64  377    0.492    0.030
0.026                  <==
> i386
> 
> i386
> # ntpq -p
>      remote           refid      st t when poll reach   delay   offset
jitter
> ==========================================================
> ====================
> xPPS(1)          .PPS.            0 l   14   16  377    0.000    0.001
0.002
> oGPS_NMEA(1)     .GPS.            0 l    8   16  377    0.000    0.000
0.002
> +2001:470:e930:2  .PPS.            1 u   30   64  377    0.519    0.008
0.020          <==pi
> 
> BBB
> # ntpq -p
>      remote           refid      st t when poll reach   delay   offset
jitter
> ==========================================================
> ====================
> oPPS(0)          .PPS.            0 s   32   32  377    0.000    0.000
0.004
> *GPS_NMEA(0)     .GPS.            0 s    7    8  377    0.000  -47.650
39.504
> +2001:470:e930:2 .PPS.            1 u   28   32  377    0.682   -0.164
0.043         <== pi
> +2001:470:e930:7 .GPS.            1 u   44   64  377    0.963   -0.197
1.771        <==
> i386
> 
> The offset being about half of the path variance for an asymmetric path is
> expected ntp behavior. All of these systems are looking at the same PPS
> pulse, though the i386 is seeing it through an RS232 driver delay so it
has an
> additional 1.3us fudge factor.
> 
> In any case, it appears that there is some form of delay in getting bits
out of
> the box.
> 
> Tony





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?066b01d299f7$8336bc60$89a43520$>