From owner-freebsd-arch@freebsd.org Tue Jan 28 19:00:14 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3649223810F for ; Tue, 28 Jan 2020 19:00:14 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 486bXY0mTpz4PBm for ; Tue, 28 Jan 2020 19:00:12 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pj1-x1035.google.com with SMTP id j17so662983pjz.3 for ; Tue, 28 Jan 2020 11:00:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=zw3615vQXsIaHhJOd6fvzos0uF3HrGxVuE8Iksc6Q84=; b=h3kL8A3FHQxrOstr8uAz7Z80wrtsXVO2ennQR3Sn+lr51qMdjGjzfboDsxpPJQmGuZ 6mspOUDnYGXzDbOmJrHF6ud6DLMKZkXaiZ19qE9/tOIO5Pz9kc2oFFSpMYl51j5gNoWP 0D1riE3utisxmDkF4G13evbnHJ4soUzZ+szZp+15ybwKlfu/+VLbVoDpNHsj8v4/psOz yFXGMT79+ao9BaSENOz5VDfFSGc7oxOrAouID1Y1ljon/3rcuKAWctLGq4I1HOv8CfI1 wpqDA6FnZKdqsR0NUdSO7aQlOvNqljBqyG0U/Ggv8ES/GQ3SXHMjYud2lJU23I/yebWj a4Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=zw3615vQXsIaHhJOd6fvzos0uF3HrGxVuE8Iksc6Q84=; b=Qeiesf4oQODvXal2ldX9P14WJB+4f0gFwrsokBof6SrMNRkhr8gVU5Cc9K/RWlOrTN SUMKn6GunV9VxNGdaI78v2a0oCpz1V1wmleD9/c0IMxpJcH+fGt8sn+8vObGoE5g9pr4 W19K/ViKCrgfpBiTURFQFmlDxxtIzSK5Eq9PcceHNQjk1pHb/lfDORUf/jfLWKx4KjWh XoflCLF55dINXET7ElYt5mnVygAPx83pUntOWdIYmBlBS55YVyrEvmBFk3SVL5vHFcFg 6yzvHlaD1NiMcxbHZkBnnWaKDM2YiDCtyooY9EX73tSQQWxrVR4sVckZDq3jIJiAIQg/ iVgQ== X-Gm-Message-State: APjAAAUmI4k4Wv8lgV8VJVLYkqK2LEWtxnFsqrPzvWvVXYBWe7QnkHtv YiKhnzwSkOFBdjUaWe1IGUcAZRhc9iI= X-Google-Smtp-Source: APXvYqxx3M+LV0fUWwGlgQZ7bAei3+NBkITSIO39MJcF5lJ4pHm7fyjFv6W9lW9k7HL+kXH5SesGbQ== X-Received: by 2002:a17:90a:8a0e:: with SMTP id w14mr6506911pjn.51.1580238005636; Tue, 28 Jan 2020 11:00:05 -0800 (PST) Received: from rrcs-76-81-105-82.west.biz.rr.com (rrcs-76-81-105-82.west.biz.rr.com. [76.81.105.82]) by smtp.gmail.com with ESMTPSA id t65sm20309062pfd.178.2020.01.28.11.00.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Jan 2020 11:00:05 -0800 (PST) Date: Tue, 28 Jan 2020 09:00:03 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Peter Jeremy cc: freebsd-arch@freebsd.org Subject: Re: Minimum memory for ZFS (was Re: svn commit: r356758 - in head/usr.sbin/bsdinstall: . scripts) In-Reply-To: <20200128064206.GC18006@server.rulingia.com> Message-ID: References: <202001230207.00N274xO042659@mail.karels.net> <20200128064206.GC18006@server.rulingia.com> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 486bXY0mTpz4PBm X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jroberson-net.20150623.gappssmtp.com header.s=20150623 header.b=h3kL8A3F; dmarc=none; spf=none (mx1.freebsd.org: domain of jroberson@jroberson.net has no SPF policy when checking 2607:f8b0:4864:20::1035) smtp.mailfrom=jroberson@jroberson.net X-Spamd-Result: default: False [-2.57 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[jroberson-net.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; DMARC_NA(0.00)[jroberson.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[jroberson-net.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.77)[ipnet: 2607:f8b0::/32(-2.04), asn: 15169(-1.78), country: US(-0.05)] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jan 2020 19:00:14 -0000 On Tue, 28 Jan 2020, Peter Jeremy wrote: > On 2020-Jan-26 10:58:15 -1000, Jeff Roberson wrote: >> My proposal is this, limit ARC to some reasonable fraction of memory, say >> 1/8th, and then do the following: >> >> On expiration from arc place the pages in a vm object associated with the >> device. The VM is now free to keep them or re-use them for user memory. >> >> On miss from arc check the page cache and take the pages back if they >> exist. >> >> On invalidation you need to invalidate the page cache. > > ZFS already has a mechanism for the VM system to request that ARC be > shrunk. This was designed around the Solaris VM system and isn't a perfect > match with the FreeBSD VM system. There has been a lot of work, within > FreeBSD, over the years to improve the way this behaves but it's obviously > not perfect. One point of confusion is that FreeBSD and Solaris use > identical terms to mean different things within their VM systems. It wasn't a very good match for the Solaris VM system either. According to various accounts of people involved in the project they had intended to integrate with the page cache but ran out of time. It was meant for fileservers and it made a lot of compromises for general purpose workloads that have been fixed over time. For example fsync requiring the zil was not in the original design. This is one of those compromises. mmap and sendfile are others. There are ways to fix that as well FWIW but it requires some changes to our fault/object apis. > >> ARC already allows spilling to SSD. I don't know the particulars of the >> interface but we're essentially spilling to memory that can be reclaimed >> by the page daemon as necessary. > > This is L2ARC. ZFS expects that L2ARC refers to a fixed amount of space > that ZFS has control over. I'm unaware of any mechanism for an external > system to reclaim L2ARC space - it would require the external system to > request ZFS (via an, AFAIK, non-existent interface) to free L2ARC objects. > > ZFS only implements a single level of L2ARC so re-using L2ARC for this new > caching mechanism would also prevent the use of traditional SSD-based L2ARC > on FreeBSD. My point was only that it has hooks for moving things out of arc already. Not that this is a perfect fit. > > Since ZFS already has a mechanism for an external system to request that > ZFS release memory, I believe effort would be better expended in getting > FreeBSD to better exchange memory pressure data with ZFS via that existing > mechanism, rather than inventing a new mechanism that is intended to > achieve much the same result, whilst effectively removing a ZFS feature. It is very challenging to implement a sane global memory policy when you have a number of uncoordinated local caches making decisions. Many people have invested effort into making this system work better but still we need tuning to run properly at different memory sizes and workloads. I find this solution to be an architecturally weak compromise. If the trivially small amount of performance afforded to you by additional ARC is important it is much easier to opt in with a sysctl while everyone else enjoys a more self-tuning and dynamic system. Thanks, Jeff > > -- > Peter Jeremy >