From owner-freebsd-current@freebsd.org Thu Sep 13 15:06:10 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25B4A1090677 for ; Thu, 13 Sep 2018 15:06:10 +0000 (UTC) (envelope-from dnebdal@gmail.com) Received: from mail-yw1-xc2a.google.com (mail-yw1-xc2a.google.com [IPv6:2607:f8b0:4864:20::c2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8A3974EC7; Thu, 13 Sep 2018 15:06:09 +0000 (UTC) (envelope-from dnebdal@gmail.com) Received: by mail-yw1-xc2a.google.com with SMTP id x83-v6so1285011ywd.4; Thu, 13 Sep 2018 08:06:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QzvpVB2MHHTix4Bm2rW0OSmG1O10qQ00VM7h3Mb5j18=; b=V6XioMhyyLUoxMq8bXC+xclTYHZzkXBYodS5rkStNfIttKE9cY/6Xt9M9TIprVUY8t tYlxA/ueGhEr5Ov+1q5pzcH6dAJrGjEIe6ZNK+fNqQAibBe3rNyWRt/3E891uJ04azyd WZwmy9AyswFWwVS8C7kcgA9R2qm/oYmZY+EqzyucdcTf4geV1SgCan+FdDcaCf6aHUtx 3nSrdh5o5GzSkhRWt9j/t6vxO8mCIk0etKNms/v9T0xx17lfkGPQ2waYVSFOxJqrR5P0 9B+baUtgRvCgi4DFQqN+r6ag5rOObiqjyOreSRnDMr8LA/8tWSF85ROFJL6Rzy0CTnVb wPeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QzvpVB2MHHTix4Bm2rW0OSmG1O10qQ00VM7h3Mb5j18=; b=FFSTG3fjgpVwdNHfZq1vPV2ZlJMmHijAyq9PNysu0G6Us3+Y11cVOM+5A37juLbnU2 H7ysDVCOWcnKdl5z43MpzzyTG/E98c8syFmoLuMs6BKK3K4ds3AIIxAO6Te039/9uTY0 wBzQnUznhKpFp2uXKUZaxR1FmCi72eub0RSpzmiS6HDaFsGRxBfQotbSk1FmB0iMh3Ea lqTPKC46yQND0/ySa2oKLTHiXRmGKdBE7LHUKWogZtoClyA+tBDj7p48tCvBQtwVTu4o m0mO+540CqhPFoeddMmktIOFnO9gAKagdIuhfbuXeLSA23i0tSrTgzTeg1NsTkxqt7px /NlA== X-Gm-Message-State: APzg51CkDibvFGLyWTrQhZnHXYUcBH78ddwrJW4C5iMJlT+TXk7zMeU0 Inx15UonwMt5xn6wkJJVc7KXYskmj4KAt+7F0P8= X-Google-Smtp-Source: ANB0VdaL3h9SQKpUu1k0LONc4d4DVghgw+wUzScgSHW+a1Rp3xwxSncPJo5N8rRFwQJh6a0j3Nc3S7a8eKLeoMQbtv4= X-Received: by 2002:a0d:e707:: with SMTP id q7-v6mr3513033ywe.436.1536851168719; Thu, 13 Sep 2018 08:06:08 -0700 (PDT) MIME-Version: 1.0 References: <0cdba37f-6b47-0929-fb72-dcca7e70a05a@vangyzen.net> In-Reply-To: <0cdba37f-6b47-0929-fb72-dcca7e70a05a@vangyzen.net> From: Daniel Nebdal Date: Thu, 13 Sep 2018 16:14:32 +0200 Message-ID: Subject: Re: arc_reclaim_thread running hot To: eric@vangyzen.net Cc: Current , markj@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Sep 2018 15:06:10 -0000 On Thu, 13 Sep 2018 at 15:19, Eric van Gyzen wrote: > > This morning, I found the arc_reclaim_thread running hot on my laptop > running 12.0-ALPHA5 r338572. > > vfs.zfs.arc_max="4294967296" <-- 4 GiB > > last pid: 13288; load averages: 1.32, 1.26, 1.16 > > Mem: 456M Active, 3837M Inact, 743M Laundry, 2563M Wired, 167M Free > ARC: 1131M Total, 304M MFU, 145M MRU, 1344K Anon, 9116K Header, 671M Other > 89M Compressed, 361M Uncompressed, 4.03:1 Ratio > > 22 root -8 - 0 256K CPU2 2 309:20 99.75% > zfskern{arc_reclaim_thread} > > zfs_arc_meta_strategy is still the default of 1. > > I sampled the thread's stacks with > > for N in `jot 1000`; do procstat -kk 100101; done | grep 100101 > > and put the results here: > > https://people.freebsd.org/~vangyzen/arc_reclaim_thread_stacks.txt > > I'm happy to help debug this. Just let me know what you need. > > Eric Just to summarize, these are the top 15 active (guessing that the leftmost function is "current"): 22 arc_space_return 22 uma_dbg_free 26 uma_dbg_alloc 30 mi_switch 31 uma_zalloc_arg 33 multilist_sublist_lock 34 __mtx_unlock_flags 39 46 _sx_xlock 48 hdr_full_cons 50 uma_zfree_arg 58 aggsum_add 91 arc_adjust 156 _sx_xunlock 185 arc_evict_state And the top 15 in total (that is, no matter where in the call stack): 44 multilist_sublist_lock 44 uma_dbg_free 46 _sx_xlock 47 uma_dbg_alloc 61 arc_space_return 127 aggsum_add 139 hdr_full_cons 156 _sx_xunlock 211 uma_zfree_arg 219 uma_zalloc_arg 833 arc_evict_state 926 arc_adjust 961 arc_reclaim_thread 961 fork_exit 961 fork_trampoline (I have nothing sensible to add about the actual problem.) -- Daniel Nebdal