From owner-freebsd-current@freebsd.org Fri Dec 7 13:40:34 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BBB8130DF4D; Fri, 7 Dec 2018 13:40:34 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) by mx1.freebsd.org (Postfix) with ESMTP id D61FC8B684; Fri, 7 Dec 2018 13:40:33 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.186] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id B2D3F3B78; Fri, 7 Dec 2018 16:40:26 +0300 (MSK) Reply-To: lev@FreeBSD.org To: FreeBSD Net , FreeBSD Current From: Lev Serebryakov Subject: iflib.tx_abdicate: very strange behavior on incoming IPsec traffic (regression?) Openpgp: preference=signencrypt Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGZBBMBCABDAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEWIQT5bRygtfQxi2dLMwrqsDxYv9xHjwUCW/03kQUJDwW3xgAK CRDqsDxYv9xHjx8ZD/sFjawMpLNn0HLWPxpe0zIfmEPbl9IQLmdlb4tC7MAZf0fOUrEuyLyP W96IDlgPAgJRwMZL4B+rFq8EUzXAj0tTM7pWMUvdeNAkpHcMG79ScgzjlmFOrPD30/fgWa/v XjV0rZhHRC+V3ANZNmJ3ELZUd5vqC5KpQDrbXmVrYqYrTCF/QRmCFSspdLdEZyMfpv0D1RlP YLtNo9CNzUkZykD8hcWamCtJzwbvhSmhISAKnSjzrxi+P4o2p7HYljt5l8tfBtDJK/MmEUon aRoFxmOpZhs0N0emUt9GyyPA5LJDjr3xK8IWb+XR+cOPiR+bEO2yC+RxF7P/2VPVMzHUMHFE TI3BXyhsny9IUi0asz/Ju0i9ZPWS9etTkOguc1TvzG0gpnni5G3zo9TVtWSL+lI1VVhPy0uP oKoP9OAvmNuL4/u7zKQVRHr3cJiBHZlXIhbnZJXJzaJrqHG9vRim6rkWUQxohPpoapMJqeXu fQJ3j6d/gL5HsEgrF5UdbH4xD/05+TrGHqye3gADow48Tt++EwKiRuX7VuAQP51ZsfRRildf wPWHNVLh+i3UHy3+6Zk3JNQsYOJHR/t6ihg+t7AEdFodu1w45wSlyVDeyfp3LVjGSTL8ODoU 13U8lYVrelEihM8PDOaZWOLBasQRAHvIoEF5atV43SEle8GIU+Jdxc7BTQRSmxpLARAAtGvc H46Nj9yck+y3PkJWxVbIRm0m2s2+Jmu0GBuSGWQ+b3ywRZHj6BunJ6591gl6eAON4bMwMpws HuHl8gJL+PoDxd2UnjROVs982Bvo4dp5tohgpMmlcpH2wPtnIaK9gyh+bR6bjTAZRfsEiR2S iZpaP7BSJLQJ1dTQVYOP3C/vf4SKb5fNnJd4WAg+/3u/6kRPsN/AqQAM5+uRb2zg5+8ZWN37 CbfnBzwSeBATPSp08HHHvhfu9kUDGUBJe7yAMEkZi3YGRzdDBQud95etklaCo09HmjjiAa73 rJqI4Vo38LoIm16/lkRf7RnFu+wccHtnkC2xFREqlnjDkcUcaHMe0O2gAqB5n+BuFNVc8e81 BHzXx4342XkFXwFEX/zKFHpWUlvwZjCHygoN5UZVgLfj6todlAAz1agGc3uOW3hP3PM/C2bD mYhqEYWC/DUoOvs4i03ZmC1nbRRgLmYajrTaeYerRo9tTpLAaLXGiGuWI2wpppyJiIxpUgZF CryIav0V7YeW9oDcGkHbvmL+AJWVR0bLlcsC/7JAxS+flntdWwyujEOT875p7USXIG8qIrJ9 YYldODldS6SSTJCZ9Y/FZQag0DU/8/gYwdK7v9Gp7/ubi4+79HJNOqxKOMol0/2QHATO+o+1 exWhm5KG2um1FAkNh+WGATiqYpIwp6cAEQEAAcLBfAQYAQgAJgIbDBYhBPltHKC19DGLZ0sz CuqwPFi/3EePBQJb/TeyBQkPBbfnAAoJEOqwPFi/3EePcKcP/1u0K2RymFFhFR7Ty1TJXDNX +zIMJ2JSEXvyOoMpML7fO7KsUmLu3h8X1+iWUdjPVGLlGLHEkee4MVyrtlM6ms9io7xCFyQs Ak7Ll2Y5ENfaC4QD7LnMNgHDCOmKj122e4TtDIfIzHicw+rw78d877p0MRkKGSsa9wb0VMju zQmGwOzfHHCNxE5nY4/SU27bnvVmlHhi7MxQ4iE5pAEgXbCtO8uMpSwCc1Bgr2CwVN4MdGwT gdkvEm8R2JkJY3sI6L26fjG0/gqFzcWafoA0TKHRsiDHUSKLZEYg24DZdFViE9EktRlFJVBy s0P2T1L6b5OhqrP0mAUu6wUDUqF15icTVuiOmrVTVuyYSJwtbZB8QZW6YpOBStOyejRxU8fv coMgsW8cDQPy3hRpC95FNFRV0dnTYgFW3rnlDxerVOQcnuX7YKSyjkgG6NYPvhClJHPN9+f+ KGEQTHBhJOMVKzQL86BT4QNltyrZJa7ifU4axbffv/zQeckpVYE8BljLgSaLJBJjTkwb9+P1 ED1CLhOMUsMush/p0UCaKIjBaOUAqLLJST5cMletzUvZWFwep90Fjgj+rBXVgAjVAjhb+LpO y0HWPQbAprdk5HC7h5qEvn71HrEF52FdmkIvR3Iq4jyI9oLyelo4Imgvz6Xj4pdH88q3xvyT Mhy9KnIJmKI9 Organization: FreeBSD Cc: olivier@cochard.me Message-ID: <94ec8936-4a66-ee1a-5a38-ee27a4bbfbee@FreeBSD.org> Date: Fri, 7 Dec 2018 16:40:19 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="MWllwvVeaX9POWIOzC225fEgXC2kMGAFF" X-Rspamd-Queue-Id: D61FC8B684 X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.995,0]; NEURAL_HAM_SHORT(-0.97)[-0.967,0]; ASN(0.00)[asn:24940, ipnet:148.251.0.0/16, country:DE]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Dec 2018 13:40:34 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF Content-Type: multipart/mixed; boundary="S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464"; protected-headers="v1" From: Lev Serebryakov Reply-To: lev@FreeBSD.org To: FreeBSD Net , FreeBSD Current Cc: olivier@cochard.me Message-ID: <94ec8936-4a66-ee1a-5a38-ee27a4bbfbee@FreeBSD.org> Subject: iflib.tx_abdicate: very strange behavior on incoming IPsec traffic (regression?) --S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable (I'm not sure, that it is exactly "bug" or "defect" and want to I've found very strange behavior of 13-CURRENT system I210 (igb) interfaces and enabled "dev.igb.X.iflib.tx_abdicate". I'm measuring "router" performance with BSDRP's "equilibrium" script (thank you, Oliver, for this great tool!). It generates traffic to route with pkt-gen and try to find packet rate / bandwidth with binary search. I'm testing simple UDP traffic via physical connection, without any GIF/GRE and other pseudo-interfaces. Router pass UDP traffic from igb1 to igb0, and this traffic is for ONLY ONE IP:PORT pair, as I'm imitating edge router for small network where only one host will receive huge amounts of traffic (i.e. torrent-box). When I enable "dev.igb.X.iflib.tx_abdicate" on both igb1 (inbound) and igb0 (outbound) interface, packet per second become a little better. So far so good. Now I'm throwing IPsec into mix. All incoming traffic is tunneled with IPsec policy, with aes-128-gcm encryption. And with IPsec tx_abdicate makes thing much worse and much more unstable. There is results without tx_abdicate: 480Mbit/s, 182Kpps And it is results with tx_abdicate: 352MBit/s, 85Kpps. And what is worse, "equilibrium" script starts to see unstable packet rate. Without tx_abdicate or without IPsec process of searching for "maximum" packet rate is very stable: each next measurement in binary search looks like previous, there is no big jumps and found "equilibrium" rate is very close to "maximum seen", and overloaded router shows rate smaller than equilibrium one). But with both "tx_abdicate" and IPsec it looks like (please, note, that overloaded router shows much better rate than not-overloaded): Benchmark tool using equilibrium throughput method - Benchmark mode: Throughput (pps) for Router - UDP load =3D 18B, IPv4 packet size=3D46B, Ethernet frame size=3D60B - Link rate =3D 1488 Kpps - Tolerance =3D 0.01 Iteration 1 - Offering load =3D 744 Kpps - Step =3D 372 Kpps - Measured forwarding rate =3D 120 Kpps - Forwared rate too low, forcing OLOAD=3DFWRATE and STEP=3DFWRATE/2 Iteration 2 - Offering load =3D 120 Kpps - Step =3D 60 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 81 Kpps Iteration 3 - Offering load =3D 60 Kpps - Step =3D 60 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 60 Kpps Iteration 4 - Offering load =3D 90 Kpps - Step =3D 30 Kpps - Trend =3D increasing - Measured forwarding rate =3D 84 Kpps Iteration 5 - Offering load =3D 75 Kpps - Step =3D 15 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 75 Kpps Iteration 6 - Offering load =3D 82 Kpps - Step =3D 7 Kpps - Trend =3D increasing - Measured forwarding rate =3D 81 Kpps Iteration 7 - Offering load =3D 85 Kpps - Step =3D 3 Kpps - Trend =3D increasing - Measured forwarding rate =3D 85 Kpps Iteration 8 - Offering load =3D 86 Kpps - Step =3D 1 Kpps - Trend =3D increasing - Measured forwarding rate =3D 86 Kpps Estimated Equilibrium Ethernet throughput=3D 86 Kpps (maximum value seen:= 120 Kpps) --=20 // Lev Serebryakov --S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464-- --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlwKeEhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+WaBAAzoidVr14yYmjqhkbcfQCKRbgkyL1jAPsGszSfJ5GtG5eDmpdh0LwiNit 5pSrCRBj+xw2g2kjT31f0h64Git1qkbEOjAFlkCKxLDQTMIlz+C/C5eAENrmABHe b6noILKPKWMt7mc0uwr/Hg/Pze6uMMv+lE3zpKeHCEZWU8g107w/6x4OBnaNkbSa Un1wy08heHimS6p60DlOkTvYYfvw6bA4X8q3WC8CfzChg9zJEG5FoJMpjA5juX3E VzeGhCKUTeRmVUcGhbguL/r1IXbymqcKbBzOmU/FNNSe25LJ7/BlY1x49e3lDKj6 AVh04n3r8M3pXmjYCPJgljkUT1EbL+6inut+kUyoV4+2cTX7SLNMMHxhrVY3/eGV OzwYW75Y8TWMXv+1TaAk5/VCiCPcZ6G5+o2DnvdAMAUdRWwV4caDSmj3ujFoLdfu 23cvMl4ARqZthOIkJX34ylbY8Hw/Jqunao70WQPVFwUigjq/ksB9mldP7X9Kty1R iE0B0aPN7jS3QZjH5S8wOCXIJY/kjI5GYPYQZtqzgzIrqubQ3j4F8Y8UHk82oXTL VUqvsf2auod6vVogKOp5AiENSj4URxt1lIJ3sVNsEjcAeBjCvrd/QUkWKLbo2hUZ ZvExi1mlNQG0tYzeaOeNFsytHzHZ1QSjRuu7EF7I8MeeVRpuvSU= =SXkp -----END PGP SIGNATURE----- --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF--