From owner-freebsd-pf@freebsd.org Sun Aug 7 14:20:25 2016 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A316BB1670; Sun, 7 Aug 2016 14:20:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-io0-x22b.google.com (mail-io0-x22b.google.com [IPv6:2607:f8b0:4001:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A6F61955; Sun, 7 Aug 2016 14:20:25 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-io0-x22b.google.com with SMTP id q83so338578116iod.1; Sun, 07 Aug 2016 07:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-transfer-encoding; bh=UYWCxp5yU7qfLIm6sHEWuF8W8vDCjNZ4lynM1EqfHoQ=; b=BzGtnGMBF4lekfBqyUWsnGpqo3gjwvDSIg+HDDqnWHYehCkW94C40pPg3hjpkGn6F4 v7B9izcxdRbNVeVALwUNTjpBmr+7yGMbr0tFICBXPrsa4G8r2iWjV48v/DRqwIZQ1R08 kSO1tz/oOjCsT0HWMm6fctLBy/X/Zfbzg7Uac+IXFIZTnhNBoRR7iHATnxX2KlBbP8im QBmzIAi78gYX9chkjSsnfNMQsVeo50X43gnIve/S0fYhlj0UjxHzQuFF4UCQIBzKKqYs tKXKRW8Xg2oJ1cKZ3jGvJo6JtsF+8wPlZ2ADr+wKEGbc8t8fMVH1EeD8svaRnupFCfl3 olgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-transfer-encoding; bh=UYWCxp5yU7qfLIm6sHEWuF8W8vDCjNZ4lynM1EqfHoQ=; b=h2RENc0O74pzBR4hb8KzaqOepSZm/c9cgTBzTEbGOTFvVYATrTCXmqMgYF05VywOtg tmGSwLbRjiLvs2kF68cXZhq2NKxSL75MvZIJp4gO5CSZlR+puPm5Hckm2Fmvz3VHJBTg 5RB9sjzamczwXGuFgS05VlhHRZfyrUHfIoyM0PXY9X1u3CF2JmIEjNrtZI13Mm5UbErP FhyK1cKLPwwJiDabb8W2GzP7cMPjpyRUawbXDUosoCoOW2fbX5i3LzjooR6ng9VQxi7L RpC/ClGmvlp0g2X5zJQS4ZKoy+S87HbHq+x0rGOsuJTSleuGzCvt84RL0kHIZDHGHmj/ ml+Q== X-Gm-Message-State: AEkoouss2jT+puua5d6le+CJSjcUhhGycSnGoLk6daYL7j0sH1TFob3yxmmT7oIr6Fnidw== X-Received: by 10.107.134.145 with SMTP id q17mr90082517ioi.25.1470579623747; Sun, 07 Aug 2016 07:20:23 -0700 (PDT) Received: from [10.0.10.3] (cpe-24-165-196-54.neo.res.rr.com. [24.165.196.54]) by smtp.googlemail.com with ESMTPSA id o15sm10427392ith.2.2016.08.07.07.20.22 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 07 Aug 2016 07:20:22 -0700 (PDT) Message-ID: <57A743A8.10005@gmail.com> Date: Sun, 07 Aug 2016 10:20:24 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: Firewalling jails and lo0 References: <20160806155411.GA5289@len-t420.klaas> <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net> <20160807082651.GA87754@box-hlm-03.niklaas.eu> In-Reply-To: <20160807082651.GA87754@box-hlm-03.niklaas.eu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2016 14:20:25 -0000 Niklaas Baudet von Gersdorff wrote: > Bjoern A. Zeeb [2016-08-06 20:02 +0000] : > >> I am curious about this. Can you give me an (obfuscated) example? (if >> you want in private email) > > -- $ jls -v > JID Hostname Path > Name State > CPUSetID > IP Address(es) > [...] > 7 mx.box-hlm-03.niklaas.eu /usr/local/jails/smtp1 > smtp1 ACTIVE > 8 > 10.3.8.1 > fd16:dcc0:f4cc:3::8:1 > [...] > 24 proxy1.box-hlm-03.niklaas.eu /usr/local/jails/proxy1 > proxy1 ACTIVE > 5 > 10.3.2.1 > 10.77.2.1 > fd16:dcc0:f4cc:3::2:1 > fd16:dcc0:f4cc:77::2:1 > [...] > -- > > -- $ ifconfig lo1 > lo1: flags=8049 metric 0 mtu 16384 > options=600003 > [...] > inet 10.3.8.1 netmask 0xffff0000 > inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64 > [...] > inet 10.3.2.1 netmask 0xffff0000 > inet 10.77.2.1 netmask 0xffff0000 > inet6 fd16:dcc0:f4cc:3::2:1 prefixlen 64 > inet6 fd16:dcc0:f4cc:77::2:1 prefixlen 64 > [...] > nd6 options=21 > -- > > The following is a /full/ output: > > -- $ ifconfig lo0 > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > -- > > So, as you can see, the jails only have IP addresses on lo1 and > none of them has one on lo0. To make that clear: > > -- $ jexec smtp1 ifconfig > [...] > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > nd6 options=21 > lo1: flags=8049 metric 0 mtu 16384 > options=600003 > inet 10.3.8.1 netmask 0xffff0000 > inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64 > nd6 options=21 > [...] > -- > > In my pf.conf I have the following. This is a simplified extract: > > -- /etc/pf.conf > 1 ext_if = vtnet0 > 2 jail_if = lo1 > 3 > 4 table persist > 5 table persist > 6 > 7 set skip on lo0 > 8 > 9 nat on $ext_if from { } to any -> > 10 > 11 block log all > 12 > 13 pass out all keep state > 14 > 15 pass in on $jail_if proto tcp from to port { } > -- > > As you can see I have a principal block in line 11, and skip is > set on lo0 solely. That said, I block on lo1. Because of this, > I pass on lo1 in line 19. I thought this is necessary. > > However, here comes the thing: Although the jails have IP addresses > attached to lo1 only, I can see traffic like the following: > > -- $ tcpdump -nettti lo0 host 10.3.2.1 > 00:00:00.023424 AF IPv4 (2), length 64: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [S], seq 4205430985, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 349909592 ecr 0], length 0 > 00:00:00.000064 AF IPv4 (2), length 64: 10.3.8.1.9025 > 10.3.2.1.51096: Flags [S.], seq 3921176095, ack 4205430986, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3273771227 ecr 349909592], length 0 > 00:00:00.000023 AF IPv4 (2), length 56: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [.], ack 1, win 1275, options [nop,nop,TS val 349909592 ecr 3273771227], length 0 > -- > > -- $ tcpdump -nettti lo0 host fd16:dcc0:f4cc:3::8:1 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes > 00:00:00.000000 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [S], seq 3339315349, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 352469079 ecr 0], length 0 > 00:00:00.000035 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [S.], seq 3726000680, ack 3339315350, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 306734766 ecr 352469079], length 0 > 00:00:00.000044 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 1, win 1274, options [nop,nop,TS val 352469079 ecr 306734766], length 0 > 00:00:05.060320 AF IPv6 (28), length 107: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [P.], seq 1:32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 31 > 00:00:00.000113 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [F.], seq 32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 0 00:00:00.000025 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 33, win 1273, options [nop,nop,TS val 352474140 ecr 306739827], length 0 00:00:00.000413 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [F.], seq 1, ack 33, win 1274, options [nop,nop,TS val 352474140 ecr 306739827], length 0 > -- > > As you can see, this is on lo0 although the jails don't have an > IP address on it. That said, restricting traffic on lo1 doesn't > make any sense because the jails use lo0 anyway. > >> Are these ::1 connections, link-local addresses (unlikely as they should >> not be visible to jails), or full IP? > > As you can see, they are full IP. > >> And what’s the routing table entry in the base system for them? > > Have a look at the following output of netstat (I removed some lines and cells): > > -- $ netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default < > UGS vtnet0 > 10.0.0.0/8 link#4 U tap0 > 10.3.2.1 link#3 UH lo1 > 10.3.8.1 link#3 UH lo1 > 10.77.2.1 link#3 UH lo1 > 127.0.0.1 link#2 UH lo0 > > Internet6: > Destination Gateway Flags Netif Expire > ::/96 ::1 UGRS lo0 > default < > UGS vtnet0 > ::1 link#2 UH lo0 > ::ffff:0.0.0.0/96 ::1 UGRS lo0 > < > link#1 U vtnet0 > < > link#1 UHS lo0 > fd16:dcc0:f4cc:3::/64 link#3 U lo1 > fd16:dcc0:f4cc:3::1 link#4 UHS lo0 > fd16:dcc0:f4cc:3::2:1 link#3 UHS lo0 > fd16:dcc0:f4cc:3::8:1 link#3 UHS lo0 > fd16:dcc0:f4cc:77::/64 link#3 U lo1 > fd16:dcc0:f4cc:77::2:1 link#3 UHS lo0 > fe80::/10 ::1 UGRS lo0 > fe80::%vtnet0/64 link#1 U vtnet0 > fe80::< >%vtnet0 link#1 UHS lo0 > fe80::%lo0/64 link#2 U lo0 > fe80::1%lo0 link#2 UHS lo0 > fe80::%tap0/64 link#4 U tap0 > fe80::< >%tap0 link#4 UHS lo0 > ff01::%vtnet0/32 < >%vtnet0 U vtnet0 > ff01::%lo0/32 ::1 U lo0 > ff01::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1 > ff01::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0 > ff02::/16 ::1 UGRS lo0 > ff02::%vtnet0/32 < >%vtnet0 U vtnet0 > ff02::%lo0/32 ::1 U lo0 > ff02::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1 > ff02::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0 > -- > >> especially, do they have any IP address assigned to lo0 in them at all? > > No, they don't. > > Niklaas I believe the loopback interface lo1 needs 127.0.0.0/8 ip address to enable loopback functionally, and the ip address has to be a different sub-net. IE 127.0.10.1 for lo1 while the hosts lo0 uses 127.0.0.1