From owner-freebsd-current@freebsd.org Sat Jun 3 05:01:44 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC592B82505 for ; Sat, 3 Jun 2017 05:01:44 +0000 (UTC) (envelope-from 0100015c6c5499e9-e0158287-b6e0-44ae-81ba-d1715fb28927-000000@amazonses.com) Received: from a8-56.smtp-out.amazonses.com (a8-56.smtp-out.amazonses.com [54.240.8.56]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD70A84BE0 for ; Sat, 3 Jun 2017 05:01:44 +0000 (UTC) (envelope-from 0100015c6c5499e9-e0158287-b6e0-44ae-81ba-d1715fb28927-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1496466103; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=X9AR5LiS5pcZ4tmyoYuZHotUsWnCXAbYVo3KgzEGdmw=; b=p1jFa/rVIr+hpqMgOfycmgYH/M0cafO3dRYsIe8oZouOb+dwL6aNsywyNGkyQEDR 4ba4JJY7zb5Wa0mPNtLLWaxH0iw7NkkGpxVJv3ipKqLc452RgCFii4FSmqfrtV+Itfd L0wQ0NRk+LFG/YMNTHOJDWsv10W+iZoOcGL1EuHU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1496466103; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=X9AR5LiS5pcZ4tmyoYuZHotUsWnCXAbYVo3KgzEGdmw=; b=RYvFQeRZ5uM1iz0Qg/74gqxyJYEVj7IQMCBNNPcqxHDiAQ8fKyTxKiBwYz7WiTuA 4PXKAVIxuXQc5aTCxnsDTyK9XCEMyAvjvOCBts4X2iyb76JuEHCoLEepYP0ba9tWhUM WkadJt5LFMJowCvr7jqwb/EJhuRpia8x4mHEXgrA= Subject: Re: NFS client perf. degradation when SCHED_ULE is used (was when SMP enabled) To: Rick Macklem , "freebsd-current@freebsd.org" References: Cc: Andriy Gapon , "cem@freebsd.org" , "jeff@freebsd.org" , Ryan Stone From: Colin Percival Message-ID: <0100015c6c5499e9-e0158287-b6e0-44ae-81ba-d1715fb28927-000000@email.amazonses.com> Date: Sat, 3 Jun 2017 05:01:42 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2017.06.03-54.240.8.56 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Mailman-Approved-At: Sat, 03 Jun 2017 11:18:48 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jun 2017 05:01:45 -0000 On 05/28/17 13:16, Rick Macklem wrote: > cperciva@ is running a highly parallelized buuildworld and he sees better > slightly better elapsed times and much lower system CPU for SCHED_ULE. > > As such, I suspect it is the single threaded, processes mostly sleeping waiting > for I/O case that is broken. > I suspect this is how many people use NFS, since a highly parallelized make would > not be a typical NFS client task, I think? Running `make buildworld -j36` on an EC2 "c4.8xlarge" instance (36 vCPUs, 60 GB RAM, 10 GbE) with GENERIC-NODEBUG, ULE has a slight edge over 4BSD: GENERIC-NODEBUG, SCHED_4BSD: 1h14m12.48s real 6h25m44.59s user 1h4m53.42s sys 1h15m25.48s real 6h25m12.20s user 1h4m34.23s sys 1h13m34.02s real 6h25m14.44s user 1h4m09.55s sys 1h13m44.04s real 6h25m08.60s user 1h4m40.21s sys 1h14m59.69s real 6h25m53.13s user 1h4m55.20s sys 1h14m24.00s real 6h24m59.29s user 1h5m37.31s sys GENERIC-NODEBUG, SCHED_ULE: 1h13m00.61s real 6h02m47.59s user 26m45.89s sys 1h12m30.18s real 6h01m39.97s user 26m16.45s sys 1h13m08.43s real 6h01m46.94s user 26m39.20s sys 1h12m18.94s real 6h02m26.80s user 27m39.71s sys 1h13m21.38s real 6h00m46.13s user 27m14.96s sys 1h12m01.80s real 6h02m24.48s user 27m18.37s sys Running `make buildworld -j2` on an E2 "m4.large" instance (2 vCPUs, 8 GB RAM, ~ 500 Mbps network), 4BSD has a slight edge over ULE on real and sys time but is slightly worse on user time: GENERIC-NODEBUG, SCHED_4BSD: 6h29m25.17s real 7h2m56.02s user 14m52.63s sys 6h29m36.82s real 7h2m58.19s user 15m14.21s sys 6h28m27.61s real 7h1m38.24s user 14m56.91s sys 6h27m05.42s real 7h1m38.57s user 15m04.31s sys GENERIC-NODEBUG, SCHED_ULE: 6h34m19.41s real 6h59m43.99s user 18m8.62s sys 6h33m55.08s real 6h58m44.91s user 18m4.31s sys 6h34m49.68s real 6h56m03.58s user 17m49.83s sys 6h35m22.14s real 6h58m12.62s user 17m52.05s sys Note that in both cases there is lots of idle time (although far more in the -j36 case); this is partly due to a lack of parallelism in buildworld, but largely due to having /usr/obj mounted on Amazon EFS. These differences all seem within the range which could result from cache effects due to threads staying on one CPU rather than bouncing around; so whatever Rick is tripping over, it doesn't seem to be affecting these tests. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid