From owner-freebsd-drivers@freebsd.org Tue Mar 27 09:54:29 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2455CF67565 for ; Tue, 27 Mar 2018 09:54:29 +0000 (UTC) (envelope-from oleksandr.kononov@intel.com) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "orsmga104.jf.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83B8F7A8AC for ; Tue, 27 Mar 2018 09:54:28 +0000 (UTC) (envelope-from oleksandr.kononov@intel.com) X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Mar 2018 02:53:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,366,1517904000"; d="c'?scan'208,217";a="27881244" Received: from irsmsx104.ger.corp.intel.com ([163.33.3.159]) by fmsmga008.fm.intel.com with ESMTP; 27 Mar 2018 02:53:16 -0700 Received: from irsmsx101.ger.corp.intel.com ([169.254.1.176]) by IRSMSX104.ger.corp.intel.com ([169.254.5.171]) with mapi id 14.03.0319.002; Tue, 27 Mar 2018 10:53:12 +0100 From: "Kononov, Oleksandr" To: "freebsd-drivers@freebsd.org" CC: "Vanco, Juraj" Subject: FreeBSD 11.1 contigfree performance issue Thread-Topic: FreeBSD 11.1 contigfree performance issue Thread-Index: AdPFrddNkK/KuVv6RIG+MvPYgkscuw== Date: Tue, 27 Mar 2018 09:53:11 +0000 Message-ID: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjI5YTlkMWItNWU0Yy00NTU5LTljZjMtNGQ4YjUwN2M2MDZhIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjIuNS4xOCIsIlRydXN0ZWRMYWJlbEhhc2giOiJ4NjFUdUJxWGtIbk40Q0l0eHR2RnV2Z0poUHkyKys4QXd0OWZtaElTMnhScWEySmQ0VjFLMmhRQ1U3c1wvVWZRMiJ9 dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: multipart/mixed; boundary="_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_" MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2018 09:54:29 -0000 --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I am using FreeBSD 11.1 -RELEASE-amd64 running on a single 32 core CPU and = am having issues with contigmalloc performance. Timing the function using rdtsc shows that it uses up on average about 10 m= illion cycles on that function along. Using the same code, FreeBSD and timing method I ran it on anther machine o= n two CPU's with a total of 32 cores. This gave about 12 thousand cycles on that function. Digging through the source code (on the single CPU) I found that smp_targeted_tlb_shootdown function in /usr/src/sys/x86/x86/mp_x86.c cause the majority of performance hit due to some cores remaining in a paus= ed state longer after the interrupt was send to them. I attached a sample code and Makefile in this email. Steps to recreate (and show rdtsc cycles): $ make $ kldload ./test.ko $ dmesg If anyone has any idea what is the cause of this issue, it would be greatly= appreciated. -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact = the sender and delete all copies. --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: application/octet-stream; name="Makefile" Content-Description: Makefile Content-Disposition: attachment; filename="Makefile"; size=317; creation-date="Tue, 27 Mar 2018 09:51:01 GMT"; modification-date="Tue, 27 Mar 2018 09:20:17 GMT" Content-Transfer-Encoding: base64 IyBOb3RlOiBJdCBpcyBpbXBvcnRhbnQgdG8gbWFrZSBzdXJlIHlvdSBpbmNsdWRlIHRoZSA8YnNk Lmttb2QubWs+IG1ha2VmaWxlIGFmdGVyIGRlY2xhcmluZyB0aGUgS01PRCBhbmQgU1JDUyB2YXJp YWJsZXMuCgojIERlY2xhcmUgTmFtZSBvZiBrZXJuZWwgbW9kdWxlCktNT0QgPSAgIHRlc3QKIyBF bnVtZXJhdGUgU291cmNlIGZpbGVzIGZvciBrZXJuZWwgbW9kdWxlClNSQ1MgPSAgIHRlc3QuYwoK IyBJbmNsdWRlIGtlcm5lbCBtb2R1bGUgbWFrZWZpbGUKIyAvdXNyL3NyYy9zaGFyZS9tay9ic2Qu a21vZC5tawouaW5jbHVkZSA8YnNkLmttb2QubWs+Cgo= --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: text/plain; name="test.c" Content-Description: test.c Content-Disposition: attachment; filename="test.c"; size=1523; creation-date="Tue, 27 Mar 2018 09:50:54 GMT"; modification-date="Tue, 27 Mar 2018 09:20:26 GMT" Content-Transfer-Encoding: base64 I2luY2x1ZGUgPHN5cy9wYXJhbS5oPgojaW5jbHVkZSA8c3lzL21vZHVsZS5oPgojaW5jbHVkZSA8 c3lzL2tlcm5lbC5oPgojaW5jbHVkZSA8c3lzL3N5c3RtLmg+CiNpbmNsdWRlIDxzeXMvbWFsbG9j Lmg+CgpzdGF0aWMgdm9pZCBydW5UZXN0KHZvaWQpOwpNQUxMT0NfREVDTEFSRShNX0ZPTyk7Ck1B TExPQ19ERUZJTkUoTV9GT08sICJtZm9vIiwgImZvbyBtZW0iKTsKCi8qIFRoZSBmdW5jdGlvbiBj YWxsZWQgYXQgbG9hZC91bmxvYWQuICovCnN0YXRpYyBpbnQgZXZlbnRfaGFuZGxlcihzdHJ1Y3Qg bW9kdWxlICptb2R1bGUsIGludCBldmVudCwgdm9pZCAqYXJnKSB7CiAgICBpbnQgZSA9IDA7IC8q IEVycm9yLCAwIGZvciBub3JtYWwgcmV0dXJuIHN0YXR1cyAqLwoKICAgIHN3aXRjaCAoZXZlbnQp IHsKICAgIGNhc2UgTU9EX0xPQUQ6CglydW5UZXN0KCk7CiAgICAgICAgYnJlYWs7CiAgICBjYXNl IE1PRF9VTkxPQUQ6CglwcmludGYoIkRvbmVcbiIpOwoJdXByaW50ZigiRG9uZVxuIik7CiAgICAg ICAgYnJlYWs7CiAgICBkZWZhdWx0OgogICAgICAgIGUgPSBFT1BOT1RTVVBQOyAvKiBFcnJvciwg T3BlcmF0aW9uIE5vdCBTdXBwb3J0ZWQgKi8KICAgICAgICBicmVhazsKICAgIH0KCiAgICByZXR1 cm4oZSk7Cn0KCnN0YXRpYyB2b2lkIHJ1blRlc3QoKSB7CgoJY29uc3QgaW50IHBhZ2VTaXplID0g NDA5NjsKICAgICAgICB1bnNpZ25lZCBsb25nIHNpemU7CiAgICAgICAgdm9pZCAqcHRyID0gTlVM TDsKCXVuc2lnbmVkIGxvbmcgbG9uZyB0MSwgdDI7CgogICAgICAgIGZvcihzaXplID0gMTsgc2l6 ZSA8PSAocGFnZVNpemUgLTEpOyBzaXplIDw8PSAxKSB7CgoJCXQxID0gcmR0c2MoKTsKICAgICAg ICAgICAgICAgIHB0ciA9IGNvbnRpZ21hbGxvYyhzaXplLCBNX0ZPTywgTV9XQUlUT0ssIDAsIH4w LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBhZ2VTaXplLCAwKTsKCQl0MiA9 IHJkdHNjKCk7CgkJcHJpbnRmKCJjb250aWdtYWxsb2MgY3ljbGVzOiAlbGx1XG4iLCB0Mi10MSk7 CgogICAgICAgICAgICAgICAgaWYocHRyID09IE5VTEwpCiAgICAgICAgICAgICAgICAgICAgICAg IHJldHVybjsKCgkJdDEgPSByZHRzYygpOwogICAgICAgICAgICAgICAgY29udGlnZnJlZShwdHIs IHNpemUsIE1fRk9PKTsKCQl0MiA9IHJkdHNjKCk7CgkJcHJpbnRmKCJjb250aWdmcmVlIGN5Y2xl czogJWxsdVxuXG4iLCB0Mi10MSk7CgkJdXByaW50ZigiTG9vcCBjb21wbGV0ZVxuIik7CiAgICAg ICAgfQp9CgovKiBUaGUgc2Vjb25kIGFyZ3VtZW50IG9mIERFQ0xBUkVfTU9EVUxFLiAqLwpzdGF0 aWMgbW9kdWxlZGF0YV90IHRlc3RfY29uZiA9IHsKICAgICJ0ZXN0IiwgICAgLyogbW9kdWxlIG5h bWUgKi8KICAgICBldmVudF9oYW5kbGVyLCAgLyogZXZlbnQgaGFuZGxlciAqLwogICAgIE5VTEwg ICAgICAgICAgICAvKiBleHRyYSBkYXRhICovCn07CgpERUNMQVJFX01PRFVMRSh0ZXN0LCB0ZXN0 X2NvbmYsIFNJX1NVQl9EUklWRVJTLCBTSV9PUkRFUl9NSURETEUpOwo= --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_--