From owner-freebsd-jail@freebsd.org Thu Dec 21 20:25:17 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 157F3E881FE; Thu, 21 Dec 2017 20:25:17 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::7:9a96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B528F76B5A; Thu, 21 Dec 2017 20:25:16 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2jmc3pxtzDl2; Thu, 21 Dec 2017 21:24:48 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: performance issue within VNET jail Message-Id: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> Date: Thu, 21 Dec 2017 21:24:47 +0100 To: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:25:17 -0000 Hi [ I did recently migrate my servers from bare metal to cloud instances = (OpenStack at OVH) ] [ FreeBSD 11.1-STABLE #0 r327055 = ] My setup is as follows and didn't change for the last couple of years: extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail Downloading a file (by wget) at the host is around 30 MB/s, and an = example tcpdump at extIF0 looks as follows: 19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], = proto TCP (6), length 8680) remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> = 0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 8628: HTTP 19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP When downloading the very same file within a VIMAGE jail the performance = drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at = exitIF0 looks as follows: 19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val = 1007182125 ecr 4141429], length 2896: HTTP 19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq = 1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr = 4141429], length 1448: HTTP 19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val = 1007182175 ecr 4141629], length 2896: HTTP 19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq = 2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr = 4141629], length 1448: HTTP 19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val = 1007182225 ecr 4141829], length 2896: HTTP A tcpdump at epairXa looks comparable. I did reduce all MTU settings at the involved interfaces from their = initial settings (1490) to an experimental setting of 1400, just to be = on the save side, to no avail. (FYI: I did have to reduce from 1500 to = 1490 to please IPSec after migration from bare metal to cloud = infrastructure.) Then, I did test the following settings found in the Net, to no avail = either: sysctl net.inet.tcp.tso=3D0 sysctl net.link.bridge.pfil_onlyip=3D0 sysctl net.link.bridge.pfil_bridge=3D0 sysctl net.link.bridge.pfil_member=3D0 sysctl net.add_addr_allfibs=3D0 I do have to admit that I am lost here, and that I cannot think about = what is going wrong. The last download I did try at my old severs has = been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I = did move my infrastructure from bare metal to cloud, thus I cannot test = anymore if my old servers would have shown that performance issue in the = meantime. Thus any feedback is highly recommended! Thanks in advance and regards, Michael From owner-freebsd-jail@freebsd.org Thu Dec 21 20:39:53 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B7EDE892FE; Thu, 21 Dec 2017 20:39:53 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A754677654; Thu, 21 Dec 2017 20:39:52 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from [192.168.228.1] (ptr-8ripyyhtmq100mi0yvj.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:2419:4e02:e137:9435:e3a1:1a8f]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id B482237C13; Thu, 21 Dec 2017 21:39:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1513888789; bh=6hL79YC2NK2PptCdyIOzYzmY3O5K/hx2/NTMQ/Rse0Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=OWnNGkyVj3idYVW9FX7ICcU03qHT2ys39emr96EaxRUb8uvQuw7fPK6abXz+EExqd l/JoJ/MU1u9SP2wfTfSbNUxfAjmUIEwKeN30lZ+jDvucb5CeA1HeFvyp+jaBkaUEdV CZjWBonk0/5qn/N9Feb8N2bv2oYlQfo2pm/tGcWE= From: "Kristof Provost" To: "Michael Grimm" Cc: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org Subject: Re: performance issue within VNET jail Date: Thu, 21 Dec 2017 21:39:48 +0100 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:39:53 -0000 On 21 Dec 2017, at 21:24, Michael Grimm wrote: > > I do have to admit that I am lost here, and that I cannot think about > what is going wrong. The last download I did try at my old severs has > been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and > I did move my infrastructure from bare metal to cloud, thus I cannot > test anymore if my old servers would have shown that performance issue > in the meantime. > > Thus any feedback is highly recommended! > Can you try turning off TSO? (`ifconfig $ifname -tso`) There have been issues with pf and TSO checksums, which looked a lot like this (i.e. bad TCP performance). Those problems should be fixed, but this is easy to test. Regards, Kristof From owner-freebsd-jail@freebsd.org Thu Dec 21 20:50:28 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D87B5E89DF4; Thu, 21 Dec 2017 20:50:28 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A4D8777E0C; Thu, 21 Dec 2017 20:50:28 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2kL2420czDnb; Thu, 21 Dec 2017 21:50:18 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Thu, 21 Dec 2017 21:50:17 +0100 Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> To: Kristof Provost X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:50:28 -0000 Kristof Provost wrote: >=20 > On 21 Dec 2017, at 21:24, Michael Grimm wrote: >> I do have to admit that I am lost here, and that I cannot think about = what is going wrong. The last download I did try at my old severs has = been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I = did move my infrastructure from bare metal to cloud, thus I cannot test = anymore if my old servers would have shown that performance issue in the = meantime. >>=20 >> Thus any feedback is highly recommended! > Can you try turning off TSO? (`ifconfig $ifname -tso`) >=20 > There have been issues with pf and TSO checksums, which looked a lot = like this (i.e. bad TCP performance). Those problems should be fixed, = but this is easy to test. >=20 I did try it, but without success.=20 This only worked for the external interface, though. Both epairX = interfaces didn't accept that command: ifconfig: -tso: Invalid argument I did mention that I previously tried "sysctl net.inet.tcp.tso=3D0". = That shoukld do the same, right? Thanks and regards, Michael From owner-freebsd-jail@freebsd.org Thu Dec 21 21:03:35 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4EFCE8AA49; Thu, 21 Dec 2017 21:03:35 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D494789AD; Thu, 21 Dec 2017 21:03:35 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from [192.168.228.1] (ptr-8ripyyhtmq100mi0yvj.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:2419:4e02:e137:9435:e3a1:1a8f]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id A78FC37C85; Thu, 21 Dec 2017 22:03:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1513890213; bh=xvo9zMCVIMPojS8zyhAPkTetnm0MPO1558zfbZsWy0s=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=lNNwIGEDS0F1GHY6Bj5lZ9DTjQ7vgyQUNL2ZnugnkGnzXUc8vAHXo6RzgybgZAqZu enje2OtN10mwPmcEdN6QKYriRUSkHi/1FuHX2eFdv8ETu3UomOyQwuhWJu0IDbA/Rs z8eebIPX0dB9cqVCtS9ipiTBYAbt9lxdew8BCqKE= From: "Kristof Provost" To: "Michael Grimm" Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Subject: Re: performance issue within VNET jail Date: Thu, 21 Dec 2017 22:03:32 +0100 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:03:35 -0000 On 21 Dec 2017, at 21:50, Michael Grimm wrote: > Kristof Provost wrote: >> >> On 21 Dec 2017, at 21:24, Michael Grimm wrote: > >>> I do have to admit that I am lost here, and that I cannot think >>> about what is going wrong. The last download I did try at my old >>> severs has been some weeks ago. Ever since I did upgrade FreeBSD >>> 11.1-STABLE, and I did move my infrastructure from bare metal to >>> cloud, thus I cannot test anymore if my old servers would have shown >>> that performance issue in the meantime. >>> >>> Thus any feedback is highly recommended! > >> Can you try turning off TSO? (`ifconfig $ifname -tso`) >> >> There have been issues with pf and TSO checksums, which looked a lot >> like this (i.e. bad TCP performance). Those problems should be fixed, >> but this is easy to test. >> > > I did try it, but without success. > Hmm. I’ve got no ideas at the moment. I run a very similar setup (although on CURRENT), and see no performance issues from my jails. Can you test a performance test without pf? Perhaps from the local LAN for example? That should help narrow it down a bit, at least. Regards, Kristof From owner-freebsd-jail@freebsd.org Thu Dec 21 21:42:44 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E11BDE8CF93; Thu, 21 Dec 2017 21:42:44 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [IPv6:2001:41d0:401:2100::5:8a0e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA5847A13F; Thu, 21 Dec 2017 21:42:44 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2lV844m7z5t; Thu, 21 Dec 2017 22:42:24 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Thu, 21 Dec 2017 22:42:22 +0100 Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> To: Kristof Provost X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:42:45 -0000 Kristof Provost wrote > On 21 Dec 2017, at 21:50, Michael Grimm wrote: >> Kristof Provost wrote: >>> Can you try turning off TSO? (`ifconfig $ifname -tso`) >>>=20 >>> There have been issues with pf and TSO checksums, which looked a lot = like this (i.e. bad TCP performance). Those problems should be fixed, = but this is easy to test. >> I did try it, but without success. > Hmm. I=E2=80=99ve got no ideas at the moment. I run a very similar = setup (although on CURRENT), and see no performance issues from my = jails. > Can you test a performance test without pf? Perhaps from the local LAN = for example? That should help narrow it down a bit, at least. Well I prepared on of my webservers running at hostB/jailX to serve a = sample file for local downloading tests: 1) hostA wget from hostB/jailX sample file: about 30 MB/s 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s 3) hostB wget from hostB/jailX sample file: about 190 MB/s 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) = include passing two firewalls, one at each host. BUT: Both hosts are = connected via an IPSec tunnel, and that's esp not tcp. Can anyone draw conclusions from this test?=20 I cannot ;-) Thanks and regards, Michael From owner-freebsd-jail@freebsd.org Thu Dec 21 21:49:15 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3545AE8D5BF; Thu, 21 Dec 2017 21:49:15 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BC6837A447; Thu, 21 Dec 2017 21:49:14 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBLLmveh015114 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 21 Dec 2017 22:48:57 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBLLmrlS003447 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 22 Dec 2017 04:48:53 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm , Kristof Provost References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org From: Eugene Grosbein Message-ID: <5A3C2C42.6060904@grosbein.net> Date: Fri, 22 Dec 2017 04:48:50 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:49:15 -0000 22.12.2017 4:42, Michael Grimm wrote: > Well I prepared on of my webservers running at hostB/jailX to serve a sample file for local downloading tests: > > 1) hostA wget from hostB/jailX sample file: about 30 MB/s > 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s > 3) hostB wget from hostB/jailX sample file: about 190 MB/s > 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s > > Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) include passing two firewalls, one at each host. BUT: Both hosts are connected via an IPSec tunnel, and that's esp not tcp. > > Can anyone draw conclusions from this test? > I cannot ;-) Make sure and double check that your ESP packets do not get fragmented. From owner-freebsd-jail@freebsd.org Thu Dec 21 21:59:49 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13CE4E8E0B3; Thu, 21 Dec 2017 21:59:49 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BCBAD7ABAB; Thu, 21 Dec 2017 21:59:48 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2lt21D2MzDsY; Thu, 21 Dec 2017 22:59:38 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: <5A3C2C42.6060904@grosbein.net> Date: Thu, 21 Dec 2017 22:59:37 +0100 Cc: Kristof Provost , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> To: Eugene Grosbein X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:59:49 -0000 > On 21. Dec 2017, at 22:48, Eugene Grosbein wrote: >=20 > 22.12.2017 4:42, Michael Grimm wrote: >=20 >> Well I prepared on of my webservers running at hostB/jailX to serve a = sample file for local downloading tests: >>=20 >> 1) hostA wget from hostB/jailX sample file: about 30 MB/s >> 2) hostA/jailY wget from hostB/jailX sample file: about 30 = MB/s >> 3) hostB wget from hostB/jailX sample file: about 190 MB/s >> 4) hostB/jailY wget from hostB/jailX sample file: about 190 = MB/s >>=20 >> Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) = include passing two firewalls, one at each host. BUT: Both hosts are = connected via an IPSec tunnel, and that's esp not tcp. >>=20 >> Can anyone draw conclusions from this test?=20 >> I cannot ;-) >=20 > Make sure and double check that your ESP packets do not get = fragmented. Hmm, I do not know how to achieve that. May the following tcpdump = excerpts answer your question, or do you want me to look somewhere else? At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp = -vv" 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], = proto ESP (50), length 140) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe699), length 120 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341151 IP (tos 0x0, ttl 64, id 40483, offset 0, flags [none], = proto ESP (50), length 140) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe69a), length 120 22:52:42.341169 IP (tos 0x0, ttl 53, id 64312, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341238 IP (tos 0x0, ttl 53, id 64314, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 At hostB the same dump looks like: 22:52:42.463511 IP (tos 0x0, ttl 53, id 41153, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaa8), length 104 22:52:42.463518 IP (tos 0x0, ttl 53, id 41155, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaa9), length 104 22:52:42.463593 IP (tos 0x0, ttl 53, id 41157, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaaa), length 104 22:52:42.463601 IP (tos 0x0, ttl 53, id 41159, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaab), length 104 22:52:42.463673 IP (tos 0x0, ttl 53, id 41161, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaac), length 104 Thanks and regards, Michael >=20 >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Thu Dec 21 22:21:06 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76B50E8F1F1; Thu, 21 Dec 2017 22:21:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CB317B59D; Thu, 21 Dec 2017 22:21:05 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBLMKr5Z015312 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 21 Dec 2017 23:20:54 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBLMKooX012573 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 22 Dec 2017 05:20:50 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> Cc: Kristof Provost , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org From: Eugene Grosbein Message-ID: <5A3C33BF.9050902@grosbein.net> Date: Fri, 22 Dec 2017 05:20:47 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 22:21:06 -0000 22.12.2017 4:59, Michael Grimm wrote: >> Make sure and double check that your ESP packets do not get fragmented. > > > Hmm, I do not know how to achieve that. May the following tcpdump excerpts answer your question, or do you want me to look somewhere else? > > At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp -vv" > > 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], proto ESP (50), length 140) > hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5fe699), length 120 > 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags [none], proto ESP (50), length 100) > hostB > hostA: ip-proto-50 It shows non-zero offsets, so your ESP packets *are* fragmented. I guess, this is the reason of your problems as fragmented ESP packets are known to cause problems due to different reasons. Simpliest way to avoid such issues is to decrease MTU of IPSEC tunnel and/or TCP MSS so that incapsulated ESP packets do not get fragmented. From owner-freebsd-jail@freebsd.org Thu Dec 21 22:35:05 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDF5DE8FF42; Thu, 21 Dec 2017 22:35:05 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [87.98.149.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9129C7BD4B; Thu, 21 Dec 2017 22:35:04 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2mfn4J5GzD0; Thu, 21 Dec 2017 23:34:57 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: <5A3C33BF.9050902@grosbein.net> Date: Thu, 21 Dec 2017 23:34:56 +0100 Cc: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <998F52B1-F07C-4A2D-ABB5-3F86D7D4BD09@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> <5A3C33BF.9050902@grosbein.net> To: Eugene Grosbein X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 22:35:06 -0000 Eugene Grosbein wrote: > 22.12.2017 4:59, Michael Grimm wrote: >>> Make sure and double check that your ESP packets do not get = fragmented. >>=20 >>=20 >> Hmm, I do not know how to achieve that. May the following tcpdump = excerpts answer your question, or do you want me to look somewhere else? >>=20 >> At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp = -vv" >>=20 >> 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags = [none], proto ESP (50), length 140) >> hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe699), length 120 >> 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags = [none], proto ESP (50), length 100) >> hostB > hostA: ip-proto-50 >=20 > It shows non-zero offsets, so your ESP packets *are* fragmented. > I guess, this is the reason of your problems as fragmented ESP packets = are known to cause problems > due to different reasons. Simpliest way to avoid such issues is to = decrease MTU of IPSEC tunnel > and/or TCP MSS so that incapsulated ESP packets do not get fragmented. Well, you already helped me out with IPSEC very recently, and I already = did decrease my MTU from 1500 to 1490. That increased my tunnel = performance dramatically, already. Thanks, I will decrease MTU further. BUT: In this thread I did report that I already had decreased MTU for = testing purposes on all involved interfaces down to 1400 to no avail, = and that my performance issue is regarding downloads within VNET jails = using TCP, not ESP. The very same external interfaces do not show a = performance drop if connected via ESP tunnel, but when trying to = download files from the internet, and only when the download is started = within a VNET jail. At the host downloads are only limited by the = bandwidth provided by the hosting company. BUT: It might well be that I did completely misunderstood your reply = instead ;-) Thanks and regards, Michael From owner-freebsd-jail@freebsd.org Fri Dec 22 19:12:11 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D37EE8407E; Fri, 22 Dec 2017 19:12:11 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E81B63F3E; Fri, 22 Dec 2017 19:12:10 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a] (p2003008C2E03DC01C8F88A2BF09D2A5A.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3J6B0y89z3DS; Fri, 22 Dec 2017 20:12:02 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Fri, 22 Dec 2017 20:11:59 +0100 Cc: Kristof Provost , Eugene Grosbein Content-Transfer-Encoding: quoted-printable Message-Id: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> To: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 19:12:11 -0000 Kristof Provost wrote: > I run a very similar setup (although on CURRENT), and see no = performance issues from my jails. In utter despair I did upgrade one server to CURRENT (#327076) today, = but that hasn't been successful :-( Ok, right now I do know: (#) there is *no* performance loss (TCP) when: (-) fetching files from outside through PF/extIF to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to jail via bridge (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) to host (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail (#) there is a *dramatic* performance loss (TCP) when: (-) fetching files from outside through PF/extIF via bridge to = jail (#) I did try to tweak the following settings *without* success: (-) sysctl net.inet.tcp.tso=3D0=20 (-) sysctl net.link.bridge.pfil_onlyip=3D0 (-) sysctl net.link.bridge.pfil_bridge=3D0 (-) sysctl net.link.bridge.pfil_member=3D0=20 (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, = bridge, epairXs (-) deactivating "scrub in all" and "scrub out on $extIF all = random-id" in /etc/pf.conf (-) setting "set require-order yes" and "set require-order no" = in /etc/pf.conf [1] [1] I do see more a lot of out-of-order packages within a jail "netstat = -s -p tcp" after those slow downloads, but not after downloads via IPSEC = tunnel from partner host. That leads me to the conclusions: (#) the bridge is not to blame (#) it's either the PF/NATing or something else, right? Thanks for your suggestions so far, but I am lost here. Any ideas? Regards, Michael From owner-freebsd-jail@freebsd.org Fri Dec 22 20:16:07 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F2C7E871C5; Fri, 22 Dec 2017 20:16:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B57B650B2; Fri, 22 Dec 2017 20:16:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBMKFr99024657 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 22 Dec 2017 21:15:53 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBMKFgPO076905 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 23 Dec 2017 03:15:42 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> From: Eugene Grosbein Message-ID: <5A3D67EC.6010907@grosbein.net> Date: Sat, 23 Dec 2017 03:15:40 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 20:16:07 -0000 23.12.2017 2:11, Michael Grimm wrote: > Kristof Provost wrote: > >> I run a very similar setup (although on CURRENT), and see no performance issues from my jails. > > In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-( > > Ok, right now I do know: > > (#) there is *no* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail > > (#) there is a *dramatic* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF via bridge to jail > > (#) I did try to tweak the following settings *without* success: > > (-) sysctl net.inet.tcp.tso=0 > (-) sysctl net.link.bridge.pfil_onlyip=0 > (-) sysctl net.link.bridge.pfil_bridge=0 > (-) sysctl net.link.bridge.pfil_member=0 > (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs > (-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf > (-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1] > > [1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host. > > That leads me to the conclusions: > > (#) the bridge is not to blame > (#) it's either the PF/NATing or something else, right? > > Thanks for your suggestions so far, but I am lost here. Any ideas? It seems to me some kind of bug in the PF. I personally never tried it, I use ipfw and it works just fine. Maybe, you should try to switch to it too, at least for a test.