Date: Sat, 15 Jun 2019 11:35:59 +0200 From: "Kristof Provost" <kp@FreeBSD.org> To: "Li-Wen Hsu" <lwhsu@freebsd.org> Cc: freebsd-testing@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: FreeBSD CI Weekly Report 2019-06-09 Message-ID: <F747EF77-9881-46A1-BC31-DF136BD75112@FreeBSD.org> In-Reply-To: <CAKBkRUyN=GN5TO44QTaeVS_rjcY9tvghACMyimgf4WMT9XsLXQ@mail.gmail.com> References: <CAKBkRUyN=GN5TO44QTaeVS_rjcY9tvghACMyimgf4WMT9XsLXQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12 Jun 2019, at 16:49, Li-Wen Hsu wrote: > * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ > * Same as amd64: > * sys.netinet.socket_afinet.socket_afinet_bind_zero > * Others: > * sys.netpfil.pf.forward.v6 > * sys.netpfil.pf.forward.v4 > * sys.netpfil.pf.set_tos.v4 I’ve finally gotten around to taking a look at this, and it appears to not be a pf problem. forward:v4 already fails at its sanity check, before it configures pf. It creates a vnet jail, telling it to route traffic through, and then we run a sanity check with pft_ping.py. Scapy tries to resolve the MAC address of the gateway (jail, 192.0.2.1). The jail replies, but scapy never picks up the reply, so the traffic looks like this: 13:19:29.953468 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.2 tell 192.0.2.1, length 28 13:19:29.953572 02:be:b4:57:9f:0b > 00:a0:98:b2:48:59, ethertype ARP (0x0806), length 42: Reply 192.0.2.2 is-at 02:be:b4:57:9f:0b, length 28 13:19:32.082843 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 52: 192.0.2.1 > 198.51.100.3: ICMP echo request, id 0, seq 0, length 18 The jail doesn’t forward the broadcast ICMP echo request and the test fails. My current guess is that it’s related to bpf. It’s interesting to note that it fails on i386, but succeeds on amd64. -- Kristof From owner-freebsd-stable@freebsd.org Sat Jun 15 13:34:58 2019 Return-Path: <owner-freebsd-stable@freebsd.org> Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B33515C79AB; Sat, 15 Jun 2019 13:34:58 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84178759F9; Sat, 15 Jun 2019 13:34:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 29656D936; Sat, 15 Jun 2019 13:34:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [172.26.30.5] (vpn.bally-wulff.de [212.144.118.13]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id BD6E31F626; Sat, 15 Jun 2019 15:34:53 +0200 (CEST) From: "Kristof Provost" <kp@FreeBSD.org> To: "Li-Wen Hsu" <lwhsu@freebsd.org> Cc: freebsd-testing@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: FreeBSD CI Weekly Report 2019-06-09 Date: Sat, 15 Jun 2019 15:34:53 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: <356AE1B0-B3D7-4BE2-A052-7A1F996E3F37@FreeBSD.org> In-Reply-To: <F747EF77-9881-46A1-BC31-DF136BD75112@FreeBSD.org> References: <CAKBkRUyN=GN5TO44QTaeVS_rjcY9tvghACMyimgf4WMT9XsLXQ@mail.gmail.com> <F747EF77-9881-46A1-BC31-DF136BD75112@FreeBSD.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 84178759F9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>; List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 15 Jun 2019 13:34:58 -0000 On 15 Jun 2019, at 11:35, Kristof Provost wrote: > On 12 Jun 2019, at 16:49, Li-Wen Hsu wrote: >> * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ >> * Same as amd64: >> * sys.netinet.socket_afinet.socket_afinet_bind_zero >> * Others: >> * sys.netpfil.pf.forward.v6 >> * sys.netpfil.pf.forward.v4 >> * sys.netpfil.pf.set_tos.v4 > > I’ve finally gotten around to taking a look at this, and it appears > to not be a pf problem. forward:v4 already fails at its sanity check, > before it configures pf. > > It creates a vnet jail, telling it to route traffic through, and then > we run a sanity check with pft_ping.py. > Scapy tries to resolve the MAC address of the gateway (jail, > 192.0.2.1). The jail replies, but scapy never picks up the reply, so > the traffic looks like this: > > 13:19:29.953468 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype ARP > (0x0806), length 42: Request who-has 192.0.2.2 tell 192.0.2.1, length > 28 > 13:19:29.953572 02:be:b4:57:9f:0b > 00:a0:98:b2:48:59, ethertype ARP > (0x0806), length 42: Reply 192.0.2.2 is-at 02:be:b4:57:9f:0b, length > 28 > 13:19:32.082843 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype IPv4 > (0x0800), length 52: 192.0.2.1 > 198.51.100.3: ICMP echo request, id > 0, seq 0, length 18 > > The jail doesn’t forward the broadcast ICMP echo request and the > test fails. > > My current guess is that it’s related to bpf. It’s interesting to > note that it fails on i386, but succeeds on amd64. > I’ve done a little dtracing, and I think that points at bpf too: #!/usr/sbin/dtrace -s fbt:kernel:bpf_buffer_uiomove:entry { tracemem(arg1, 1500, arg2); stack(); } Results in: 1 49539 bpf_buffer_uiomove:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: ce 0e 05 5d 17 ea 00 00 2a 00 00 00 2a 00 00 00 ...]....*...*... 10: 12 00 ff ff ff ff ff ff 02 fd 10 30 e6 0a 08 06 ...........0.... 20: 00 01 08 00 06 04 00 01 00 a0 98 b2 48 59 c0 00 ............HY.. 30: 02 01 00 00 00 00 00 00 c0 00 02 02 ce 0e 05 5d ...............] 40: 60 ea 00 00 2a 00 00 00 2a 00 00 00 12 00 00 a0 `...*...*....... 50: 98 b2 48 59 02 fd 10 30 e6 0b 08 06 00 01 08 00 ..HY...0........ 60: 06 04 00 02 02 fd 10 30 e6 0b c0 00 02 02 00 a0 .......0........ 70: 98 b2 48 59 c0 00 02 01 ..HY.... kernel`bpfread+0x137 kernel`dofileread+0x6d kernel`kern_readv+0x3b kernel`sys_read+0x48 kernel`syscall+0x2b4 0xffc033b7 So, we see the ARP request through bpf, but we don’t see the reply, despite tcpdump capturing it. I have no idea how that’d happen, so I’d very much like someone more familiar with bpf to take a look at this problem. Regards, Kristof
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F747EF77-9881-46A1-BC31-DF136BD75112>