From owner-freebsd-arm@freebsd.org Mon Mar 6 02:03:54 2017 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69AF2CF92F8 for ; Mon, 6 Mar 2017 02:03:54 +0000 (UTC) (envelope-from tony@tndh.net) Received: from express.tndh.net (express.tndh.net [IPv6:2001:470:e930:1240:20d:56ff:fe04:4c0a]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3DFDA1350; Mon, 6 Mar 2017 02:03:54 +0000 (UTC) (envelope-from tony@tndh.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=tndh.net; s=dkim; h=Subject:Content-Transfer-Encoding:Content-Type:MIME-Version:Message-ID:Date:Cc:To:From; bh=VfZJe7eNUH62g4zMD8ZTE/q7mTVBNYn3s7c+Jgd54Yw=; b=AWgk0I5zE4HlgCMEJfupaNrlfoppuYr05uPX8owmLwQgCv5nLFIjsrYdELSxmc4HS0dvEaHBa8KEwTmCU70XsLvg0Znezaxsq0MDPChr20KemWyF2rJxVxx2lIelHHDMLvwRMv6pfz0ehimwI+6FnrAEc8sztaFM0CXoX/Se+evNsvvH; Received: from express.tndh.net ([2001:470:e930:1240:20d:56ff:fe04:4c0a] helo=eaglet) by express.tndh.net with esmtp (Exim 4.72 (FreeBSD)) (envelope-from ) id 1cki02-0006QP-HQ; Sun, 05 Mar 2017 18:03:52 -0800 From: "Tony Hain" To: Cc: "'Ian Lepore'" Date: Sun, 5 Mar 2017 18:03:46 -0800 Message-ID: <01c701d2961d$e3dd6100$ab982300$@tndh.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AdKWFnDH3u1k5S7URoq/cuLJiik7jA== Content-Language: en-us X-SA-Exim-Connect-IP: 2001:470:e930:1240:20d:56ff:fe04:4c0a X-SA-Exim-Mail-From: tony@tndh.net X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on express.tndh.net X-Spam-Level: X-Spam-Status: No, score=-1.0 required=4.5 tests=ALL_TRUSTED,RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Subject: BBB asymmetric stack? X-SA-Exim-Version: 4.2 X-SA-Exim-Scanned: Yes (on express.tndh.net) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2017 02:03:54 -0000 FreeBSD 12.0-CURRENT #0 r314118M: Wed Feb 22 22:49:19 PST 2017 Trying to calibrate the ntp feed on the BBB I was suspecting some problem with the dmtpps driver, but then realized that network asymmetry would be more likely to explain the persistent offset. It would appear that there is an asymmetry within the stack on the am335x based system. ping6 Rpi -> BBB 10 packets transmitted, 10 received, 0% packet loss, time 9006ms rtt min/avg/max/mdev = 0.503/0.567/0.642/0.047 ms ping6 BBB -> Rpi 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.878/0.967/1.141/0.134 ms Those are on adjacent ports on the same 100mbps switch. --- ping6 Fbsd-8 -> BBB 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.599/0.734/1.041/0.112 ms ping6 BBB -> Fbsd-8 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 0.855/0.970/1.816/0.296 ms Those are one router hop apart. --- Even localhost appears to indicate a problem in the FreeBSD-arm side of this: pi# ping6 ::1 PING ::1(::1) 56 data bytes 64 bytes from ::1: icmp_seq=1 ttl=64 time=0.220 ms 64 bytes from ::1: icmp_seq=2 ttl=64 time=0.222 ms 64 bytes from ::1: icmp_seq=3 ttl=64 time=0.221 ms i386# ping6 ::1 PING6(56=40+8+8 bytes) ::1 --> ::1 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.185 ms 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.135 ms 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.129 ms bbb# ping6 ::1 PING6(56=40+8+8 bytes) ::1 --> ::1 16 bytes from ::1, icmp_seq=0 hlim=64 time=0.672 ms 16 bytes from ::1, icmp_seq=1 hlim=64 time=0.477 ms 16 bytes from ::1, icmp_seq=2 hlim=64 time=0.464 ms I realize the loopback ping by itself is not a good indicator of an asymmetry problem, but FreeBSD 8 on i386 systems ping::1 at ~ 130us, while all the FreeBSD 10/11 VMs (Hyper-V & VMware) and the pi ping ::1 at ~220us. The issue is that the time is always longer when the BBB initiates the ping, which would imply some transmit delay that does not exist when responding to an incoming ping, but it could be simply getting the reply back to user space which only happens in the BBB initiate case. ==== The ntp perspective might shed some light on which direction. pi pi# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================ == GPS_NMEA(0) .GPS. 0 s 342d 16 0 0.000 -29.245 0.000 oPPS(0) .PPS. 0 s 6 16 377 0.000 0.000 0.002 *172.20.0.18 .PPS. 1 u 12 64 377 0.492 0.030 0.026 <== i386 i386 # ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================ == xPPS(1) .PPS. 0 l 14 16 377 0.000 0.001 0.002 oGPS_NMEA(1) .GPS. 0 l 8 16 377 0.000 0.000 0.002 +2001:470:e930:2 .PPS. 1 u 30 64 377 0.519 0.008 0.020 <==pi BBB # ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================ == oPPS(0) .PPS. 0 s 32 32 377 0.000 0.000 0.004 *GPS_NMEA(0) .GPS. 0 s 7 8 377 0.000 -47.650 39.504 +2001:470:e930:2 .PPS. 1 u 28 32 377 0.682 -0.164 0.043 <== pi +2001:470:e930:7 .GPS. 1 u 44 64 377 0.963 -0.197 1.771 <== i386 The offset being about half of the path variance for an asymmetric path is expected ntp behavior. All of these systems are looking at the same PPS pulse, though the i386 is seeing it through an RS232 driver delay so it has an additional 1.3us fudge factor. In any case, it appears that there is some form of delay in getting bits out of the box. Tony