From owner-freebsd-current@freebsd.org Sun May 28 20:16:57 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11E3BD86348 for ; Sun, 28 May 2017 20:16:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660072.outbound.protection.outlook.com [40.107.66.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B214D1553; Sun, 28 May 2017 20:16:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0191.CANPRD01.PROD.OUTLOOK.COM (10.165.218.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Sun, 28 May 2017 20:16:54 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1124.015; Sun, 28 May 2017 20:16:54 +0000 From: Rick Macklem To: "freebsd-current@freebsd.org" CC: Andriy Gapon , "cem@freebsd.org" , "jeff@freebsd.org" , Ryan Stone , "Colin Percival" Subject: Re: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Topic: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) Thread-Index: AQHS1mmgxqkdxhAY5U2DZ8FjIm51zqIIvdIkgAFvXYY= Date: Sun, 28 May 2017 20:16:53 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0191; 7:O2ZEQIZ+AVS1CCuMktupu/2jqPa7aGpnYGW51c+WtdsEwfrU7wk5hBpDv6Cq2bzTsj5/GCFrcx92CTBej1KuU7HR6J6s5a8rn5tqk9a3Rq92ZFiJOjT1atAxFm8RiajWmwAaMOQDPCvhwpmpUo6zOqksgVQiOPg161se2Pp8552MzhHffvRPPjH0Aez+A5W/EazC1HP5ez/6ybDgKon9MYWi8aifsisUP2YSA63rIPacB/ZW/vzfjFwH5Qu3O8c32nbYv0weyZnqnSMHw1c5T4qpjagE86Io/mMyWjq2NSd9aqdfRJ7Trbh9RGV/LA3IwFZil7YNKMVEKTmePt6oSA== x-ms-traffictypediagnostic: YTXPR01MB0191: x-ms-office365-filtering-correlation-id: 32923207-d713-460e-f07a-08d4a6067ba2 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081); SRVR:YTXPR01MB0191; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(2401047)(5005006)(8121501046)(3002001)(93006095)(93001095)(10201501046)(6041248)(20161123558100)(20161123560025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123564025)(20161123555025)(6072148); SRVR:YTXPR01MB0191; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0191; x-forefront-prvs: 03218BFD9F x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39850400002)(39400400002)(39410400002)(39840400002)(39450400003)(24454002)(229853002)(86362001)(6916009)(3660700001)(2950100002)(3280700002)(7696004)(76176999)(54356999)(50986999)(8936002)(25786009)(4326008)(450100002)(2900100001)(81166006)(8676002)(189998001)(305945005)(102836003)(33656002)(74316002)(5660300001)(2906002)(74482002)(122556002)(2351001)(54906002)(77096006)(53936002)(55016002)(478600001)(9686003)(110136004)(38730400002)(6246003)(6436002)(6506006)(2501003)(5640700003)(14454004); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0191; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 28 May 2017 20:16:53.9748 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0191 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 May 2017 20:16:57 -0000 I wrote: [stuff snipped] > So, I'd say either reverting the patch or replacing it with the "obvious = change" mentioned > in the commit message will at least mostly fix the problem. "mostly fix" was probably a bit optimistic. Here's my current #s. (All cases are the same single threaded kernel build, same hardware, etc. T= he only changes are recent vs 1yr old head kernel and what is noted.) - 1yr old kernel, SMP, SCHED_ULE 94minutes - 1yr old kernel, no SMP, SCHED_ULE 111minutes - recent kernel, SMP, SCHED_4BSD 104minutes - recent kernel, no SMP, SCHED_ULE 113minutes - recent kernel, SMP, SCHED_ULE, r312426 reverted 122minutes - recent kernel, SMP, SCHED_ULE 148minutes So, reverting r312426 only gets rid of about 1/2 of the degradation. One more thing I will note is that the system CPU is higher for the cases t= hat run with lower/better elapsed times: - 1yr old kernel, SMP, SCHED_ULE 545s - 1yr old kernel, no SMP, SCHED_ULE 293s - recent kernel, no SMP, SCHED_ULE 292s - recent kernel, SMP, SCHED_ULE 466s cperciva@ is running a highly parallelized buuildworld and he sees better slightly better elapsed times and much lower system CPU for SCHED_ULE. As such, I suspect it is the single threaded, processes mostly sleeping wai= ting for I/O case that is broken. I suspect this is how many people use NFS, since a highly parallelized make= would not be a typical NFS client task, I think? There are other changes to sched_ule.c in the last year, but I'm not sure w= hich would be easy to revert and might make a difference in this case? rick ps: I've cc'd cperiva@ and he might wish to report his results. I am hoping= he does try a make without "-j" at some point.