Date: Fri, 10 Mar 2017 15:39:08 -0800 From: "Tony Hain" <tony@tndh.net> To: <freebsd-arm@freebsd.org> Cc: "'Ian Lepore'" <ian@freebsd.org> Subject: RE: BBB asymmetric stack? Message-ID: <066b01d299f7$8336bc60$89a43520$@tndh.net>
next in thread | raw e-mail | index | archive | help
Follow up: Using OWAMP (http://software.internet2.edu/owamp/) there is a clear asymmetry in the FreeBSD-12 arm stack. Measurements originating from the BBB consistently take longer than those originating in the opposite direction, and jitter is significantly worse when FreeBSD-12 arm is doing the echo. This level of error results in an offset for ntp. FreeBSD12_BBB -> ARM Cortex-A8 1.0GHz Rpi_raspberrypi_4.1.15+ -> model B ARM Cortex-A53 1.2 GHz FreeBSD8_i386a -> hw.model: Intel(R) Celeron(R) CPU 2.40GHz FreeBSD8_i386b -> hw.model: Intel(R) Pentium(R) 4 CPU 1.80GHz BBB & Rpi connected to the same switch. i386a & i386b are 2 local router hops off that switch with routing symmetry verified. Measurements done sequentially to avoid any cross impact ::::: FreeBSD12_BBB -> # owping -c 1000 -n u 2001:470:e930:1240:-i386a- Approximately 103.2 seconds until results available --- owping statistics from [2001:470:e930:2240:-BBB-]:8769 to [2001:470:e930:1240:-i386a-]:2259 --- SID: fe044c0adc6da10e3acf2a8a67086b90 first: 2017-03-10T14:02:55.463 last: 2017-03-10T14:04:38.162 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 288/500/1.74e+04 us, (err=3.02 us) one-way jitter = 100 us (P95-P50) Hops = 2 (consistently) no reordering --- owping statistics from [2001:470:e930:1240:-i386a-]:2263 to [2001:470:e930:2240:-BBB-]:8883 --- SID: fefbf6c6dc6da10e5380d38f54bbd8b6 first: 2017-03-10T14:02:55.466 last: 2017-03-10T14:04:34.525 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 350/500/1.99e+04 us, (err=3.02 us) one-way jitter = 300 us (P95-P50) Hops = 2 (consistently) no reordering FreeBSD8_i386a -> # owping -c 1000 -n u 2001:470:e930:2240:-BBB- Approximately 103.3 seconds until results available --- owping statistics from [2001:470:e930:1240:-i386a-]:10791 to [2001:470:e930:2240:-BBB-]:4212 --- SID: fefbf6c6dc6da19d78d6292c35ef1187 first: 2017-03-10T14:05:18.697 last: 2017-03-10T14:06:54.748 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 345/500/2.18e+04 us, (err=3.02 us) one-way jitter = 400 us (P95-P50) Hops = 2 (consistently) no reordering --- owping statistics from [2001:470:e930:2240:-BBB-]:4265 to [2001:470:e930:1240:-i386a-]:42582 --- SID: fe044c0adc6da19d7cfebce7630aea48 first: 2017-03-10T14:05:18.797 last: 2017-03-10T14:06:56.113 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 295/500/6.32e+04 us, (err=3.02 us) one-way jitter = 100 us (P95-P50) Hops = 2 (consistently) no reordering FreeBSD12_BBB -> # owping -c 1000 -n u 2001:470:e930:12c0:-i386b- Approximately 103.3 seconds until results available --- owping statistics from [2001:470:e930:2240:-BBB-]:8855 to [2001:470:e930:12c0:-i386b-]:2318 --- SID: fe794e70dc6da2372cea6331afc13440 first: 2017-03-10T14:07:52.407 last: 2017-03-10T14:09:29.933 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 254/400/1.42e+04 us, (err=3.02 us) one-way jitter = 100 us (P95-P50) Hops = 2 (consistently) no reordering --- owping statistics from [2001:470:e930:12c0:-i386b-]:2300 to [2001:470:e930:2240:-BBB-]:8887 --- SID: fefbf6c6dc6da237329d336cf9a12126 first: 2017-03-10T14:07:52.463 last: 2017-03-10T14:09:30.625 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 322/500/1.36e+04 us, (err=3.02 us) one-way jitter = 200 us (P95-P50) Hops = 2 (consistently) no reordering FreeBSD8_i386b -> # owping -c 1000 -n u 2001:470:e930:2240:-BBB- Approximately 103.2 seconds until results available --- owping statistics from [2001:470:e930:12c0:-i386b-]:57055 to [2001:470:e930:2240:-BBB-]:4211 --- SID: fefbf6c6dc6da2a71e6d9f99dfe5be1b first: 2017-03-10T14:09:44.324 last: 2017-03-10T14:11:26.464 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 305/500/2.57e+04 us, (err=4.02 us) one-way jitter = 200 us (P95-P50) Hops = 2 (consistently) no reordering --- owping statistics from [2001:470:e930:2240:-BBB-]:4231 to [2001:470:e930:12c0:-i386b-]:64099 --- SID: fe794e70dc6da2a7243e1ee38651110b first: 2017-03-10T14:09:44.355 last: 2017-03-10T14:11:25.457 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 254/400/2.5e+04 us, (err=4.02 us) one-way jitter = 100 us (P95-P50) Hops = 2 (consistently) no reordering FreeBSD12_BBB -> # owping -c 1000 -n u 2001:470:e930:2240:-Rpi- Approximately 103.4 seconds until results available --- owping statistics from [2001:470:e930:2240:-BBB-]:8928 to [2001:470:e930:2240:-Rpi-]:4226 --- SID: 4b031abcdc6da34676422beb5014dbd9 first: 2017-03-10T14:12:23.808 last: 2017-03-10T14:14:03.739 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 285/400/1.82e+04 us, (err=2.01 us) one-way jitter = 200 us (P95-P50) TTL not reported no reordering --- owping statistics from [2001:470:e930:2240:-Rpi-]:4236 to [2001:470:e930:2240:-BBB-]:8813 --- SID: fefbf6c6dc6da346861b906566678c71 first: 2017-03-10T14:12:23.828 last: 2017-03-10T14:14:12.996 1000 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 243/400/2.34e+04 us, (err=2.01 us) one-way jitter = 300 us (P95-P50) TTL not reported no reordering raspberrypi_4.1.15+ -> # owping -c 100 -n u 2001:470:e930:2240:-BBB- Approximately 12.9 seconds until results available --- owping statistics from [2001:470:e930:2240:-Rpi-]:8906 to [2001:470:e930:2240:-BBB-]:4214 --- SID: fefbf6c6dc6da414e9617290304594ab first: 2017-03-10T14:15:49.962 last: 2017-03-10T14:16:00.106 100 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 255/400/2.71e+03 us, (err=2.01 us) one-way jitter = 300 us (P95-P50) TTL not reported no reordering --- owping statistics from [2001:470:e930:2240:-BBB-]:4210 to [2001:470:e930:2240:-Rpi-]:8912 --- SID: 4b031abcdc6da414f059a52f753769e5 first: 2017-03-10T14:15:50.032 last: 2017-03-10T14:16:01.078 100 sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 289/400/707 us, (err=2.01 us) one-way jitter = 200 us (P95-P50) TTL not reported no reordering > -----Original Message----- > From: Tony Hain [mailto:tony@tndh.net] > Sent: Sunday, March 05, 2017 6:04 PM > To: 'freebsd-arm@freebsd.org' > Cc: 'Ian Lepore' > Subject: BBB asymmetric stack? > > FreeBSD 12.0-CURRENT #0 r314118M: Wed Feb 22 22:49:19 PST 2017 > > Trying to calibrate the ntp feed on the BBB I was suspecting some problem > with the dmtpps driver, but then realized that network asymmetry would be > more likely to explain the persistent offset. It would appear that there is an > asymmetry within the stack on the am335x based system. > > ping6 Rpi -> BBB > 10 packets transmitted, 10 received, 0% packet loss, time 9006ms rtt > min/avg/max/mdev = 0.503/0.567/0.642/0.047 ms > > ping6 BBB -> Rpi > 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip > min/avg/max/std-dev = 0.878/0.967/1.141/0.134 ms > > Those are on adjacent ports on the same 100mbps switch. > > --- > > ping6 Fbsd-8 -> BBB > 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip > min/avg/max/std-dev = 0.599/0.734/1.041/0.112 ms > > ping6 BBB -> Fbsd-8 > 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip > min/avg/max/std-dev = 0.855/0.970/1.816/0.296 ms > > Those are one router hop apart. > > --- > > Even localhost appears to indicate a problem in the FreeBSD-arm side of this: > pi# ping6 ::1 > PING ::1(::1) 56 data bytes > 64 bytes from ::1: icmp_seq=1 ttl=64 time=0.220 ms > 64 bytes from ::1: icmp_seq=2 ttl=64 time=0.222 ms > 64 bytes from ::1: icmp_seq=3 ttl=64 time=0.221 ms > > i386# ping6 ::1 > PING6(56=40+8+8 bytes) ::1 --> ::1 > 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.185 ms > 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.135 ms > 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.129 ms > > bbb# ping6 ::1 > PING6(56=40+8+8 bytes) ::1 --> ::1 > 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.672 ms > 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.477 ms > 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.464 ms > > > I realize the loopback ping by itself is not a good indicator of an asymmetry > problem, but FreeBSD 8 on i386 systems ping::1 at ~ 130us, while all the > FreeBSD 10/11 VMs (Hyper-V & VMware) and the pi ping ::1 at ~220us. The > issue is that the time is always longer when the BBB initiates the ping, which > would imply some transmit delay that does not exist when responding to an > incoming ping, but it could be simply getting the reply back to user space > which only happens in the BBB initiate case. > > ==== > The ntp perspective might shed some light on which direction. > > pi > pi# ntpq -p > remote refid st t when poll reach delay offset jitter > ========================================================== > ==================== > GPS_NMEA(0) .GPS. 0 s 342d 16 0 0.000 -29.245 0.000 > oPPS(0) .PPS. 0 s 6 16 377 0.000 0.000 0.002 > *172.20.0.18 .PPS. 1 u 12 64 377 0.492 0.030 0.026 <== > i386 > > i386 > # ntpq -p > remote refid st t when poll reach delay offset jitter > ========================================================== > ==================== > xPPS(1) .PPS. 0 l 14 16 377 0.000 0.001 0.002 > oGPS_NMEA(1) .GPS. 0 l 8 16 377 0.000 0.000 0.002 > +2001:470:e930:2 .PPS. 1 u 30 64 377 0.519 0.008 0.020 <==pi > > BBB > # ntpq -p > remote refid st t when poll reach delay offset jitter > ========================================================== > ==================== > oPPS(0) .PPS. 0 s 32 32 377 0.000 0.000 0.004 > *GPS_NMEA(0) .GPS. 0 s 7 8 377 0.000 -47.650 39.504 > +2001:470:e930:2 .PPS. 1 u 28 32 377 0.682 -0.164 0.043 <== pi > +2001:470:e930:7 .GPS. 1 u 44 64 377 0.963 -0.197 1.771 <== > i386 > > The offset being about half of the path variance for an asymmetric path is > expected ntp behavior. All of these systems are looking at the same PPS > pulse, though the i386 is seeing it through an RS232 driver delay so it has an > additional 1.3us fudge factor. > > In any case, it appears that there is some form of delay in getting bits out of > the box. > > Tony
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?066b01d299f7$8336bc60$89a43520$>