From owner-freebsd-fs@freebsd.org Tue Jul 14 15:10:31 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 440F49A1761 for ; Tue, 14 Jul 2015 15:10:31 +0000 (UTC) (envelope-from schittenden@groupon.com) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10AC8D68 for ; Tue, 14 Jul 2015 15:10:30 +0000 (UTC) (envelope-from schittenden@groupon.com) Received: by ykay190 with SMTP id y190so10404128yka.3 for ; Tue, 14 Jul 2015 08:10:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=groupon.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+9J7FmOApWdCr4Nlj+EgHIEhABHWskultPV++8sZPfg=; b=fBBj4EGea15pHe/GyHStZ+u2rdoGQKP+ZPei7Cpc0+g1f0gKxVejXhm4WYPVAePONQ JNwNjaHzssjBkqadIdjQBH/XmS7fXNVfjac794xRYeKLpaTexE3efHbSzYK0Rd7tTUg/ xVskk3XJzLQnDI+X0rSD8UmZlTI55tzEEgc6g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+9J7FmOApWdCr4Nlj+EgHIEhABHWskultPV++8sZPfg=; b=C9gl/JhQTrJE3Wn4+doUIqjv+IjO3S0BhTy/XVY3aZPm/Zg+/fd/lNscrOGn3syBKF Tjv3yjgtN6y/2khf+mqAn2nTTiSXBdAC/sgn6USo6k7EOU9rFfOTlt2zc0PieE+/Hkm4 64yLqXLoo3ZrHhJxO4p65BV9BTsbK2/FoIgDMGgy5HmOf8zNRKQnZufEI8yfadUeMPb6 lIEzr7VDA506Fm38GR+mZCXQbgts+flg92uPyIxSlQ/7mF2qjc9lmFkJ6qIOJvD5E9vL l0Ky5Zo+7Kq0jf0RFacryCkfcrTXmv0fqTgaCdagPaZIT8omVm+YyREoS8QNlp6P5NRO MtpQ== X-Gm-Message-State: ALoCoQlyYrmmwAXF/foN61Rzqj2+egEcLw/cz4BTNjnw0OMEn68r55H4Hj6XPSjVSJmlLAetYyTE MIME-Version: 1.0 X-Received: by 10.170.110.82 with SMTP id c79mr44902121ykb.113.1436886629924; Tue, 14 Jul 2015 08:10:29 -0700 (PDT) Received: by 10.13.242.7 with HTTP; Tue, 14 Jul 2015 08:10:29 -0700 (PDT) In-Reply-To: <55A4E5AB.8060909@netlabs.org> References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> Date: Tue, 14 Jul 2015 08:10:29 -0700 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Sean Chittenden To: Adrian Gschwend Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jul 2015 15:10:31 -0000 I think the reason this is not seen more often is because people frequently throw limits on the arc in /boot/loader.conf: vfs.zfs.arc_min="18G" vfs.zfs.arc_max="149G" ZFS ARC *should* not require those settings, but does currently for mixed workloads (i.e. databases) in order to be "stable". By setting fixed sizes on the ARC, UMA and ARC are much more cooperative in that they have their own memory regions to manage so this behavior is not seen as often. To be clear, however, it should not be necessary to set parameters like these in /boot/loader.conf in order to obtain consistent operational behavior. I'd be curious to know if someone running 10.2 BETA without patches is able to trigger this behavior or not. There was work done that reported helped with this between 10.1 and now. To what extent it helped, however, I don't have any advice yet. -sc On Tue, Jul 14, 2015 at 3:34 AM, Adrian Gschwend wrote: > On 14.07.15 11:26, Matthew Seaman wrote: > > > > On 07/13/15 12:58, Karl Denninger wrote: > >> Put this on your box and see if the problem goes away.... :-) > > [...] > > > I know that you, Karl, and a number of others have been advocating to > > get this patch set committed. Having now personally run into the sort > > of problems that this addresses I can say that I would very much like to > > see this go in. Conditional of course on this actually solving the > > problems I and others have been experiencing without introducing > > significant regressions elsewhere. It's only had a day's testing from me > > so far, but it's looking good. If it survives a week without the system > > locking up, I'll be convinced. > > I was the one which posted the message last year which triggered Karl to > analyze it as he saw similar issues: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019043.html > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019057.html > > Since then I run on Karls patch and never had any issue anymore. Not > that my boxes were basically unusable without the patch. > > So I'm basically hoping since then that the patch will be committed soon. > > > * The memory exhaustion effect or equivalent memory pressures can be > > triggered at will > > * The test doesn't require unfeasibly large resources to run > > * The behaviour provides a good model for real-world deployments > > > > Maybe these tests would be too large-scale to run every day in Jenkins, > > but having them available as part of, say, the release process, seems > > like a no-brainer to me. > > I wouldn't consider my setup as "unfeasibly large resources", in fact I > triggered it with a bunch of jails running on a machine and providing > various Internet-services for a small Open Source community. I was > always surprised that not more people ran into this issue as I had it > since 8.x. > > regards > > Adrian > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Sean Chittenden