Date: Fri, 7 Dec 2018 16:40:19 +0300 From: Lev Serebryakov <lev@FreeBSD.org> To: FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org> Cc: olivier@cochard.me Subject: iflib.tx_abdicate: very strange behavior on incoming IPsec traffic (regression?) Message-ID: <94ec8936-4a66-ee1a-5a38-ee27a4bbfbee@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF Content-Type: multipart/mixed; boundary="S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464"; protected-headers="v1" From: Lev Serebryakov <lev@FreeBSD.org> Reply-To: lev@FreeBSD.org To: FreeBSD Net <freebsd-net@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org> Cc: olivier@cochard.me Message-ID: <94ec8936-4a66-ee1a-5a38-ee27a4bbfbee@FreeBSD.org> Subject: iflib.tx_abdicate: very strange behavior on incoming IPsec traffic (regression?) --S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable (I'm not sure, that it is exactly "bug" or "defect" and want to I've found very strange behavior of 13-CURRENT system I210 (igb) interfaces and enabled "dev.igb.X.iflib.tx_abdicate". I'm measuring "router" performance with BSDRP's "equilibrium" script (thank you, Oliver, for this great tool!). It generates traffic to route with pkt-gen and try to find packet rate / bandwidth with binary search. I'm testing simple UDP traffic via physical connection, without any GIF/GRE and other pseudo-interfaces. Router pass UDP traffic from igb1 to igb0, and this traffic is for ONLY ONE IP:PORT pair, as I'm imitating edge router for small network where only one host will receive huge amounts of traffic (i.e. torrent-box). When I enable "dev.igb.X.iflib.tx_abdicate" on both igb1 (inbound) and igb0 (outbound) interface, packet per second become a little better. So far so good. Now I'm throwing IPsec into mix. All incoming traffic is tunneled with IPsec policy, with aes-128-gcm encryption. And with IPsec tx_abdicate makes thing much worse and much more unstable. There is results without tx_abdicate: 480Mbit/s, 182Kpps And it is results with tx_abdicate: 352MBit/s, 85Kpps. And what is worse, "equilibrium" script starts to see unstable packet rate. Without tx_abdicate or without IPsec process of searching for "maximum" packet rate is very stable: each next measurement in binary search looks like previous, there is no big jumps and found "equilibrium" rate is very close to "maximum seen", and overloaded router shows rate smaller than equilibrium one). But with both "tx_abdicate" and IPsec it looks like (please, note, that overloaded router shows much better rate than not-overloaded): Benchmark tool using equilibrium throughput method - Benchmark mode: Throughput (pps) for Router - UDP load =3D 18B, IPv4 packet size=3D46B, Ethernet frame size=3D60B - Link rate =3D 1488 Kpps - Tolerance =3D 0.01 Iteration 1 - Offering load =3D 744 Kpps - Step =3D 372 Kpps - Measured forwarding rate =3D 120 Kpps - Forwared rate too low, forcing OLOAD=3DFWRATE and STEP=3DFWRATE/2 Iteration 2 - Offering load =3D 120 Kpps - Step =3D 60 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 81 Kpps Iteration 3 - Offering load =3D 60 Kpps - Step =3D 60 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 60 Kpps Iteration 4 - Offering load =3D 90 Kpps - Step =3D 30 Kpps - Trend =3D increasing - Measured forwarding rate =3D 84 Kpps Iteration 5 - Offering load =3D 75 Kpps - Step =3D 15 Kpps - Trend =3D decreasing - Measured forwarding rate =3D 75 Kpps Iteration 6 - Offering load =3D 82 Kpps - Step =3D 7 Kpps - Trend =3D increasing - Measured forwarding rate =3D 81 Kpps Iteration 7 - Offering load =3D 85 Kpps - Step =3D 3 Kpps - Trend =3D increasing - Measured forwarding rate =3D 85 Kpps Iteration 8 - Offering load =3D 86 Kpps - Step =3D 1 Kpps - Trend =3D increasing - Measured forwarding rate =3D 86 Kpps Estimated Equilibrium Ethernet throughput=3D 86 Kpps (maximum value seen:= 120 Kpps) --=20 // Lev Serebryakov --S9ZvQ9R7s7BJmDExxQwqg3gPgpnA76464-- --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlwKeEhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4+WaBAAzoidVr14yYmjqhkbcfQCKRbgkyL1jAPsGszSfJ5GtG5eDmpdh0LwiNit 5pSrCRBj+xw2g2kjT31f0h64Git1qkbEOjAFlkCKxLDQTMIlz+C/C5eAENrmABHe b6noILKPKWMt7mc0uwr/Hg/Pze6uMMv+lE3zpKeHCEZWU8g107w/6x4OBnaNkbSa Un1wy08heHimS6p60DlOkTvYYfvw6bA4X8q3WC8CfzChg9zJEG5FoJMpjA5juX3E VzeGhCKUTeRmVUcGhbguL/r1IXbymqcKbBzOmU/FNNSe25LJ7/BlY1x49e3lDKj6 AVh04n3r8M3pXmjYCPJgljkUT1EbL+6inut+kUyoV4+2cTX7SLNMMHxhrVY3/eGV OzwYW75Y8TWMXv+1TaAk5/VCiCPcZ6G5+o2DnvdAMAUdRWwV4caDSmj3ujFoLdfu 23cvMl4ARqZthOIkJX34ylbY8Hw/Jqunao70WQPVFwUigjq/ksB9mldP7X9Kty1R iE0B0aPN7jS3QZjH5S8wOCXIJY/kjI5GYPYQZtqzgzIrqubQ3j4F8Y8UHk82oXTL VUqvsf2auod6vVogKOp5AiENSj4URxt1lIJ3sVNsEjcAeBjCvrd/QUkWKLbo2hUZ ZvExi1mlNQG0tYzeaOeNFsytHzHZ1QSjRuu7EF7I8MeeVRpuvSU= =SXkp -----END PGP SIGNATURE----- --MWllwvVeaX9POWIOzC225fEgXC2kMGAFF--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?94ec8936-4a66-ee1a-5a38-ee27a4bbfbee>