From owner-freebsd-net@freebsd.org Thu Feb 4 13:50:48 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C128A9C705; Thu, 4 Feb 2016 13:50:48 +0000 (UTC) (envelope-from honzhan@microsoft.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bn0107.outbound.protection.outlook.com [157.56.110.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 93C142C3; Thu, 4 Feb 2016 13:50:47 +0000 (UTC) (envelope-from honzhan@microsoft.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=7XURdZm9tz1byzix476KT1rU4mcxIKyGIjlUxC2DDXM=; b=HkoVMy5EBY791q+3lu0JoTmu9DZ7jGwjmWZmSPTTlUCmW1ziKIQGJEI2b1Pb4l9DoI8EO4DUrwR+dvStOqn1dp2vGDGelCyQ2ETx4RCdaNTq8442dfoHumhGIHV4mLhVg+w5td0LcZlZqfm4V8qHBsCYVt4b75YVdofMfEDpN7o= Received: from CH1PR03CA003.namprd03.prod.outlook.com (10.255.156.148) by CY1PR0301MB2089.namprd03.prod.outlook.com (10.164.2.147) with Microsoft SMTP Server (TLS) id 15.1.396.15; Thu, 4 Feb 2016 13:35:15 +0000 Received: from BN1BFFO11FD014.protection.gbl (10.255.156.132) by CH1PR03CA003.outlook.office365.com (10.255.156.148) with Microsoft SMTP Server (TLS) id 15.1.396.15 via Frontend Transport; Thu, 4 Feb 2016 13:35:15 +0000 Authentication-Results: spf=pass (sender IP is 206.191.228.180) smtp.mailfrom=microsoft.com; hob.de; dkim=none (message not signed) header.d=none;hob.de; dmarc=pass action=none header.from=microsoft.com; Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 206.191.228.180 as permitted sender) receiver=protection.outlook.com; client-ip=206.191.228.180; helo=064-smtp-out.microsoft.com; Received: from 064-smtp-out.microsoft.com (206.191.228.180) by BN1BFFO11FD014.mail.protection.outlook.com (10.58.144.77) with Microsoft SMTP Server (TLS) id 15.1.409.7 via Frontend Transport; Thu, 4 Feb 2016 13:35:14 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net (141.251.56.18) by SG2PR3002MB0105.064d.mgd.msft.net (141.251.56.17) with Microsoft SMTP Server (TLS) id 15.1.403.10; Thu, 4 Feb 2016 13:35:10 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) by SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) with mapi id 15.01.0403.016; Thu, 4 Feb 2016 13:35:10 +0000 From: Hongjiang Zhang To: "Meyer, Wolfgang" , "'freebsd-net@FreeBSD.org'" CC: "'freebsd-performance@FreeBSD.org'" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAHKtgg Date: Thu, 4 Feb 2016 13:35:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [180.170.80.203] X-MS-Office365-Filtering-Correlation-Id: 178e7cc1-5c93-4225-26e5-08d32d680392 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD014; 1:ntAtAWEK8GlRhOfrLvWWo0iNr05s703QjpHSg8sEVBLHjr4e57tdXWwKPZnyBOu4IAWJafgRZKXA+Kc0lgPysd0SAlKWHJqtX1IEP/XuoaMpZUkyHWCEIB+GssjYU802kAPLGsBjpF9ngaT+Z7EftlMGMb5LG0TXI0hp9kxkWr4m4skRjqH3C7fTmiAS009xo26h/jQ5Xl83swrN3pTUK4GgKEjIHUPJ7SMVAc3CGW8uyKoznTSnhM36ltQwfAnqLyzBZIFIaT6qElRfXZM+RuMqrabDN/J1icxjL62q26I5uqD5n0VodV/w/8sOdD4ehPTq3ZwHxgYuxSqgRRdtGg== X-Forefront-Antispam-Report: CIP:206.191.228.180; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(438002)(377454003)(3905003)(13464003)(199003)(189002)(24736003)(15975445007)(2950100001)(6116002)(97756001)(33646002)(3846002)(102836003)(47776003)(23726003)(2900100001)(92566002)(87936001)(586003)(5001960100002)(5008740100001)(76176999)(66066001)(86146001)(6806005)(1096002)(54356999)(189998001)(11100500001)(1220700001)(16796002)(50986999)(108616004)(4326007)(19580395003)(2906002)(86612001)(5003600100002)(575784001)(86362001)(10400500002)(19580405001)(5005710100001)(50466002)(46406003)(10290500002)(5001770100001)(106466001)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0301MB2089; H:064-smtp-out.microsoft.com; FPR:; SPF:Pass; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 2:yzQobEZ8/kdGTNoEndrvh42hCAgw9OjqojMd9No6JDoETWyEzXN5cN6viYGJMNvZWS52dNmqQVwoD9QYjDec/owBLUyhhoIlwYJjWZEi3Y+sT7LTcOxNc/bQ2J9nEhvKs1mIcBNYgqoPZp+USnlWDjFBySR1DyHPRzYb8umCI26voFP1J3Uq2V4YaPM1p2Cv; 3:8UsR44mbp2hsE1Oe9yacrhd3jnqM4Qm5tXT6RMyP2pJZhHiKFx6Wz5dza5snSh1vN8BmK0ECPa7s2ujbi2WAYsv0zwX0Z8VXcBXnV4Pysgn7fiMw7M9GtFMs+QWaEcrmvqP9oKhzmYzcWXRuuZTmhwtJtIVU3ILvtXjvrlND5eI6l1La6yRLziaaCyfeF3PeJp4UqfxhHQzjCdRkiON3OVThaRpsRPvlPZ5l+7yXAYQfeJmKMECe1zFUW0tdI5dW9arDkGv4+7sDG6czouUc8w==; 25:Y2SPeu6ph74MT6oSz9FllACCD5AA3OtY3CXwnNlfJA9lTKo+Xzh7oeXqhJLBh1kIiJWJs6XOipwMEUqypeQPAUNBxTIIwu5npgeFSbMpDNBGlmwZsq0l8kXx8cbky69im27sI5CkevRyjKyCS/7iKK4o+ZcAJ3d/p5LT5WLSDjQfE+T1liLchylh+bCkjXO8JC4Dk0kTEDDL60ZCIDpBAvGqbg4OJv3S+VX57uBS9HpMX/peIgGWEq/SAGw8a9htcCeEYr9PNl9VtJ2qyKIevglClKuFWQu1hHzv8nf7Cpb3Y18Q4Rwc+mASqwQLALCN0Z4fPrCfuo2DiHwHNXAzdA== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:CY1PR0301MB2089; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 20:RDCqWzyvNhSg+JxrvoKeHN/kuNwg+r4IzkBstIZFYdxaELRXVHTa5cDcSy0z+f6/P35cOeEpH6wcObwQETOhkK9XRhTotWl+ZS+zlcKtfP0kRt6loATW6QTllcdR6OQnqeTjdyjSIeiKC8DD0yb0DqyUo1qT7Xs3VT0mq4qQZCRFLlbtdE3cbQl81cP/ByAkF7HIM6ufzmBMXcos/w91I8LmyjFJiOOz2PjC0zISL6hrNbORDKKp80ZsyC9zvJ4OLsoj6PTdgwftrrrdNULkZ+k3GC2el+J3gVj1jiH7i+chrou8oZBoPOdyLyk+bzLkbRqcEozC1pp+J40fDm/xlwbYCG8S0X9k5erHEwPXrTyMhDftT3MOzbQvse6WLEw72BxOA1CWQ2Iq35KjHj481cAmkkhWMqXGtPdh4aNtWwuv/2jBZ9G0MovZY7szHXo6aIWtjnmlm175BbI+lL09bSWqzg0slvYIAuodjJSjb7wn4wYZH/a9RFGgiK4lerIUmsIfcZooK+MMJSFqXSJWW2hMql1PSFq2WSLQtZB2rWrmyN9h34PcHgzYldLD+idgzi3QvbPz5LK0UGQuZQxknhKtB1yNEZGJNOUd3aW+0e0= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(61425038)(601004)(2401047)(5005006)(13016025)(8121501046)(13018025)(3002001)(10201501046)(61426038)(61427038); SRVR:CY1PR0301MB2089; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0301MB2089; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 4:UeVcVSthzafHtLZ93teCM2JvnOVOdqPw/VhKRdM0ePuzJ8tmA3KSNoAdiHdNYqWBOGFWFPRfaT0XqnGMmUVq3QgYVzMjLIXVUVol9mYnzK0B3H+Jsed7Y5by5aaHDUMn1SNcRihsxvvJnFUxMxFG2A3wNupBdBdF16uOW7KLGfWf3tNUcyGffm+1qdfNvnianiSIdE8HWc/y5A548BxLxqEmR8GJmj7SF3qMbdlHGXwXDN57QU1tRx93/G8fffJg2AAjT3JbqAcqhyTWTut5+kdf6ZWVODhz09AbOc+DF1R3xoATuy6ddz0w8ThLESEkWXabFd4rgvrBqT1vxWM7fPjhAu9fhv/+CAbwBeyF9z8f+35h30j9Lfzkb1bf6tE5ORX7xBCYkWaiSJk0msFDhwc/FoUm37dXIbzLlyaHvbJkuru+EcRjGoBhKKdOBXLnH4OamZ/XPzG5U0x9hAPF4g== X-Forefront-PRVS: 084285FC5C X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR0301MB2089; 23:wvyZaz/+/NibWBM7DEl1u2pCBdzXgEJ9mIGWjVu?= =?us-ascii?Q?wIPmDK5dpxOJ4xUKtNEgPLpZTAn7N7naBpUsyxDzFddfc18qhvYb20FdKRb2?= =?us-ascii?Q?MMcKjvXElHYlasrRnH8l1a2tba+IJrkrTjCKPNCV+FsWxCSAArfKxRZrkP8V?= =?us-ascii?Q?6xYQTP2/niSkQsnOtPDAVGaBzcHSCohWHc0oBGTEpOHrREpuChFVAClv7Rbi?= =?us-ascii?Q?3fl8W5swdLLSZm79213eZpMRJayD/bhLePM6Oqd0zy0r4baMUME20+616PIC?= =?us-ascii?Q?8ZEKGmQX9AaoxAXvLxHiCm2VqV00IZjL6+ShqIapzae4Ka/3DXakIU9OiDdN?= =?us-ascii?Q?QkJn9f5ZzwlAZCOaz/6msyJAh9HWupyND02OCSjqaRgKJ2IhEYO+3W9uwq30?= =?us-ascii?Q?LQXYY1Dfp6OOnIMy8+awXNodLHcFNQN4KTm2GwAeGb/MqIM7QOWuACon9HFS?= =?us-ascii?Q?S33pLoNec/tKqUiZyelR5DhdKzp2DRfZem6CnQJgUgRnpVnE64IkSFNyTOCG?= =?us-ascii?Q?a/HAQ0d6yBm++eRJHwHoSu1UMcBLBhE5qjfQufv0/KrstaYRlgyqcwMWtqHG?= =?us-ascii?Q?sRScFkESOLGcTbHsVlJZfLQ0ODx2bUgaDSy9oZbzQA5P+ekfUBXdq8CCI9m3?= =?us-ascii?Q?YzeuMFyvcSBzh0Z1A6bF8TZ4QmmGX/HYchRWlJwp8GCUjSJuPW92VnhV/F+y?= =?us-ascii?Q?6SV37hE3eEw89cO1cmaf663ZUTbtjmed2Gmtl8gy17Zr1TCs5OKCNq7K7bYZ?= =?us-ascii?Q?sxE+93I2t+gOWJxx05RDSRo7JfaxvMn3EyCDQHZPxyGj5kD21R2o/m6ixBvT?= =?us-ascii?Q?QL2mSYd8RzcCPxF14p0T02uO3sWE30KgfbkXO1WgxsqQMQ8Jt7jOJtrOHAlE?= =?us-ascii?Q?vzkn3GSipYy6C5lFn4FEn68ZgYkqc6H2LtZnHAMs10iPzkC13YENQzu/Qcel?= =?us-ascii?Q?5I/leoCr6QXPc27V6HiZPUY0MzwbeHK8eekZARK45AbF+SGLEeqWftxs1F5h?= =?us-ascii?Q?JkIqkpctTcEuUrnLJIcwYrNzc/QxMt1Np2UvaHNVZ3CKapE6WzqHW4OUZmZq?= =?us-ascii?Q?TqRMVg6Jqmfs7AAWkWQco1boz1GiH639h0UYiHR0VtCaTvEGzHOSDozyXBlz?= =?us-ascii?Q?oBjA/OSUsYm1kaVJoSk1t88u6JTfQfNyiZqeoosqYxH2vcfYvBfwYyU7v01G?= =?us-ascii?Q?n9U0SFIofLzkdSNgTr/w6dziwV0lGFKzkGGTqv4lHHvQwX+ZlMWICYg3Ws+P?= =?us-ascii?Q?E6HFdM5rRvQButcVBwnvfIbdb8Dp5OethJclbaVdi5ehsPiWyBrZmpZJOkvC?= =?us-ascii?Q?0rrZyDvxp1wmod04DGctCVc2SHAkQzZ6la/f8iBCNrN+E?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 5:MEFAOzt/1NyF4xEh1bB9ouM0AptaVEQijN2ngECnXT6oyBZ8nRXs5Xux4OF75JrF8s/SpgWCGx7CFrjr4L/AA5Xj2gMwKryqJkPrvuB2iYI8/WO23rFsTAiQbI0nPW9iONFV4sH4mIyPXeJ/zOIwVA==; 24:BhMQ+p0Jz9GLDZy9zSIv2ko53fI2S30viC6Vpx5DrIFu6U6R8SWZHVx1aOMJbWo1Hoe29K8d0wIPxhydVA9J7aaxi4lTzY+nB8cQRxue8+c= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2016 13:35:14.4450 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[206.191.228.180]; Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0301MB2089 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 13:50:48 -0000 Please check whether LRO is enabled on your FreeBSD server with "ifconfig".= Linux default enables GRO (see the output of 'ethtool -k eth0'), which cov= ers LRO optimization. Thanks Hongjiang Zhang -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Meyer, Wolfgang Sent: Wednesday, February 3, 2016 9:37 PM To: 'freebsd-net@FreeBSD.org' Cc: 'freebsd-performance@FreeBSD.org' Subject: ixgbe: Network performance tuning (#TCP connections) Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://na01.safelinks.protection.outlook.com/= ?url=3Dhttps%3a%2f%2fcalomel.org%2ffreebsd_network_tuning.html%2c&data=3D01= %7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12= %7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DxsMoC%2b1ZcnoHBnPqhLUMDIr8V= LBcLejnrXgkRyDWzYc%3d https://na01.safelinks.protection.outlook.com/?url=3D= https%3a%2f%2fwww.freebsd.org%2fdoc%2fhandbook%2fconfigtuning-kernel-limits= .html%2c&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DXNqvrYfTN= zfe2btrip%2f5FoX3iTTpTSbNrDjbhtVBevo%3d https://na01.safelinks.protection.o= utlook.com/?url=3Dhttps%3a%2f%2fpleiades.ucsc.edu%2fhyades%2fFreeBSD_Networ= k_Tuning&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3D%2bQ66X%2= frnqNakX%2fSGcK08QTTrsDjUUWBHOXu6%2fOBIBNQ%3d) but most of them didn't have= any measuarable impact. Final sysctl.conf and loader.conf settings see bel= ow. Actually the only tunables that provided any improvement were identifie= d to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value= of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set = to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang : cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" : kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fww= w.hob.de%2fredirect%2fhob.html&data=3D01%7c01%7chonzhan%40064d.mgd.microsof= t.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47= %7c1&sdata=3D%2fenM%2fs72BvfP9H6CnrFeXqhrZoetovqoIMB%2bk0RFfQM%3d - Xing: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fw= ww.hob.de%2fredirect%2fxing.html&data=3D01%7c01%7chonzhan%40064d.mgd.micros= oft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db= 47%7c1&sdata=3DSi0aQmWV%2ba%2fWx5%2f6tZtOWQDj5cmq9t57DhA2h0qsa7Y%3d - LinkedIn: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2flinkedin.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3DBc0iX1u2twaazr%2fzi2wuTRqTZOk2rwtu61lNqJrui14%3d - HOBLink Mobile: https://na01.safelinks.protection.outlook.com/?url=3Dhttp= %3a%2f%2fwww.hob.de%2fredirect%2fhoblinkmobile.html&data=3D01%7c01%7chonzha= n%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f= 141af91ab2d7cd011db47%7c1&sdata=3DI1ViplWsQ2HMWM%2fLnhWRKseziJBoNlyVBj2wiWF= x1wM%3d - Facebook: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2ffacebook.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3D1WliqTPX5YN3AdvK6DzAB7yDkQmjyC3jh%2f47PZ2uU7Y%3d - Twitter: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2ftwitter.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DgaLO9OTaBis4F3IGoFY55nwMIOGPaZ0ri%2fK7N7nx7kI%3d - YouTube: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2fyoutube.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DT%2fhNogZgLHoTCOj%2fKjsSJhPDlBqyxCAD8tJj5fueiqw%3d - E-Mail: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2= fwww.hob.de%2fredirect%2fmail.html&data=3D01%7c01%7chonzhan%40064d.mgd.micr= osoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011= db47%7c1&sdata=3DQnWAw2culSVFforLRWYgXfyHaj4PyB4Hn1rj4jVQvjU%3d HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 ______= _________________________________________ freebsd-net@freebsd.org mailing list https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists.fr= eebsd.org%2fmailman%2flistinfo%2ffreebsd-net&data=3D01%7c01%7chonzhan%40064= d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91= ab2d7cd011db47%7c1&sdata=3DGU28hzKU5SSyA3hbP1PNtUNh7G5ut%2fMD6mdRmuJNkEs%3d To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"