From owner-freebsd-arch@freebsd.org Sun Jan 26 20:58:21 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B875B1FB78B for ; Sun, 26 Jan 2020 20:58:21 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 485QFm2ZyJz4VG4 for ; Sun, 26 Jan 2020 20:58:20 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pl1-x644.google.com with SMTP id ay11so3011049plb.0 for ; Sun, 26 Jan 2020 12:58:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=NU1elfIHEXkRIe0qyWt1MFvnZ4z+qoMjFq/+b8BPOnE=; b=sX5WgKq6CxsnJkJx9v9MGd7NXCb5hL1Djr6MlWTfzSyodde8NiR9GVBBWc5/Ql3k0/ L9zexI1FhZARs/NpptLtaoDwj9A9XSU1m0r1l+Jq+EoezdyR9HNKMTenHJmd8R39Tvre i5zQgnBvQlY1JWPsYBL+nn6lE9mw6r3pbxEHgNRKfel6XaBKhUwdhqznWZhVMYbbxc9C OcegNh3MC0N5sEBnD+NYBuUcA3cKZ2efwdhqn5gZFLq6jDJlHblLGugW2dn3fe0jwob/ X/pV8S8F21hLIFEG0bLOjNPGCzhd7xJ/0KftlE2cfRWaOv86FUzk6YDy6dpuAzgKSgj4 YJlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=NU1elfIHEXkRIe0qyWt1MFvnZ4z+qoMjFq/+b8BPOnE=; b=RQ7llYry4SEhCtUv1xrfSHmkeu21hPvi6/0kzirHC8dHEPSc0rwQK9zduFaqM3b+vx 1obzd10IsBr/QUnfshNtSc3twhC2nOqmsHMnEGn4RiEzt++pGkxmB0zG5sbML/wpzEgH WMJRfJC/D0MHDyMU9Ldu5p6X0V2b37OZ4n5I81I9AHIuFXHHETsGkEyccwHCV91vG6EL CHm5E326hpyN+xn0mBONZl0MB752wVhmrkxAjQ+2zcw2JD3wDnT3Y1PS76DU9HL6TDU4 ksL2Nx62h5gfQ84KJwOI6nzIMfxYMH0c41rNE3NkVGmXTS90WaPp6yHNNRjL5DMXcjxl y6GA== X-Gm-Message-State: APjAAAUyqR2D72e+VxD9B3X+rwhYs3hXSv66I/fLmPx3kdUONbdAEn3p 6iOE6FmtCMvF57JT0GXseNOy/w== X-Google-Smtp-Source: APXvYqydbVmlfODhuK71EMX5jDGTM6ulpdOBloHFbUTcwcA3W/2Vjbp7iXoqrYMQVgHoRWJ81Ichow== X-Received: by 2002:a17:90a:7781:: with SMTP id v1mr11117773pjk.108.1580072298726; Sun, 26 Jan 2020 12:58:18 -0800 (PST) Received: from rrcs-76-81-105-82.west.biz.rr.com (rrcs-76-81-105-82.west.biz.rr.com. [76.81.105.82]) by smtp.gmail.com with ESMTPSA id y14sm13134106pfe.147.2020.01.26.12.58.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 Jan 2020 12:58:18 -0800 (PST) Date: Sun, 26 Jan 2020 10:58:15 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Mike Karels cc: Ben Woods , Philip Paeps , freebsd-arch@freebsd.org, Ed Maste , Conrad Meyer Subject: Re: Minimum memory for ZFS (was Re: svn commit: r356758 - in head/usr.sbin/bsdinstall: . scripts) In-Reply-To: <202001230207.00N274xO042659@mail.karels.net> Message-ID: References: <202001230207.00N274xO042659@mail.karels.net> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 485QFm2ZyJz4VG4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jroberson-net.20150623.gappssmtp.com header.s=20150623 header.b=sX5WgKq6; dmarc=none; spf=none (mx1.freebsd.org: domain of jroberson@jroberson.net has no SPF policy when checking 2607:f8b0:4864:20::644) smtp.mailfrom=jroberson@jroberson.net X-Spamd-Result: default: False [-2.71 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[jroberson-net.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(-0.91)[ip: (-0.64), ipnet: 2607:f8b0::/32(-2.05), asn: 15169(-1.79), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; DMARC_NA(0.00)[jroberson.net]; RCPT_COUNT_FIVE(0.00)[6]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[jroberson-net.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[4.4.6.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jan 2020 20:58:21 -0000 On Wed, 22 Jan 2020, Mike Karels wrote: > I took the liberty of changing the subject line to make it stand out a > bit more. > > Ben wrote: > >> On Sat, 18 Jan 2020 at 09:16, Mike Karels wrote: > >>>> On Fri, 17 Jan 2020 at 08:21, Ben Woods wrote: >>> >>>>> Perhaps we could simply include a message on that bsdinstall >>> partitioning >>>>> mode selection screen that UFS is recommended on systems with < 4 Gb >>> RAM? >>>>> >>> >>>> I have uploaded a diff for this here: https://reviews.freebsd.org/D23224 >>> >>>> Please let me know your thoughts (comments in the phabricator review >>> would >>>> be best). >>> >>> I think this needs more discussion, preferably on this list. I am not >>> convinced that systems with as little as 4 GB should use ZFS. Conventional >>> wisdom on the FreeNAS mailing list says that 8 GB is required for ZFS, >>> and FreeNAS no longer includes UFS as an option. Conrad suggested a >>> cutoff of 16 GB; I am happier with 16 GB than 4 GB as a cutoff. Also, >>> there was mention of auto-tuning for smaller systems; I don't think that >>> has materialized yet. I'm not sure how plausible that is without knowing >>> the workload. I use ZFS on a workstation/server with 64 GB that runs 4 >>> bhyve guests that do things like buildworld. ZFS wants 63 GB for arc_max; >>> needless to say, I have a tunable set to a much lower value. If tuning >>> is required, it is unclear that ZFS is a good default. >>> >>> Mike >>> > > >> Hi everyone, > >> Before I commit phabricator review D23224, is there any final comments? > >> Particularly on these 2 lines of help-text: >> msg_partitioning_zfs_help="ZFS is recommended if you have at least 4GB RAM" >> msg_partitioning_ufs_help="UFS is recommended if you have less than 4GB of >> RAM" > >> There is some disagree about what these 2 recommendations should be. > >> 4GB was recommended by: imp, emaste, philip, eugen, dteske >> 8GB was recommended by: mike >> 16GB was recommended by: cem I have a completely different recommendation that will take someone with some knowledge of arc and a little knowledge of VM. I am happy to provide the VM side if someone else will provide the arc side and testing. If you have a reasonable understanding of ARC I suspect this could be prototyped in a weekend. Simply put, there is no reason the arc can't spill into the page cache just as the buf cache does. If you look at the actual papers on ARC/2Q/etc. you will see that their primary advantage is in limiting the impact of scans and they only provide a few percent up to 10% performance in even contrived workloads. Simply put, there is no reason to give arc all of your memory. There is a good reason to make all of your memory available for caching however. My proposal is this, limit ARC to some reasonable fraction of memory, say 1/8th, and then do the following: On expiration from arc place the pages in a vm object associated with the device. The VM is now free to keep them or re-use them for user memory. On miss from arc check the page cache and take the pages back if they exist. On invalidation you need to invalidate the page cache. ARC already allows spilling to SSD. I don't know the particulars of the interface but we're essentially spilling to memory that can be reclaimed by the page daemon as necessary. With this change the ARC would participate reasonably and we wouldn't need to talk about minimum memory for it. We have already wasted more person-hours working around this architectural misstep than would be required to address it. If anyone would like to take this up please contact me directly as I don't always read arch@. Thanks, Jeff > >> The 4GB limit seems to have the best consensus, however there was some >> debate about whether ZFS is recommended on a system with 4GB, or only >> systems with MORE THAN 4GB. > > I don't remember what everyone else wrote, but IIRC, Devin said that if > you use ZFS with 4 GB, you will soon end up with a dozen tunables set. > That doesn't sound like a recommendation for 4 GB. > >> As for the ZFS auto-tuning, I see that as being a separate discussion >> (which could ultimately change this recommendation, but shouldn't prevent >> us from committing this help text now). > > Agreed, but the lack of tuning should factor into the current recommendation. > > Mike > >> Regards, >> Ben > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >