From owner-svn-src-head@freebsd.org Tue Sep 15 15:20:41 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8024E3D6269 for ; Tue, 15 Sep 2020 15:20:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qv1-xf29.google.com (mail-qv1-xf29.google.com [IPv6:2607:f8b0:4864:20::f29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BrRkb4nwZz4FPn for ; Tue, 15 Sep 2020 15:20:39 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qv1-xf29.google.com with SMTP id di5so1855241qvb.13 for ; Tue, 15 Sep 2020 08:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VPxu8SFchIKjiN5XSqHM3N1EkkM5BiCYjn7I7gPM8XM=; b=boQitBEdyAuS6c0XKmlRIbjQVoTGYVO7nkUUbTJwM7/yyvK5Cn1b45yw2vXH5mjnJh rvB008IcdTA8JidmhaDi+hElAMHUYSRBYQPX1lYIWJ1sTqBKchtNDa0GqVnGYeQEIrGv 9PQk8lIinTq1gbAu6vTAgGh/YHNqDhk+1DY/R/LYB8gOOXQ8iPgcQqKPy/81GQfx8Pmu HmNF4gXqShcyuKtYK6ofDPEYmLeYxCmVOmNBX5lZPmjCl06SSOnM1pQcCtrK+1nD4dom LAfHhixErwtlZNNXpn3mbJX1NmpS0PluatlTxIfZ6an3soHLazq2xmrtHi2GibocVmAy VzzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VPxu8SFchIKjiN5XSqHM3N1EkkM5BiCYjn7I7gPM8XM=; b=tYyF9u+PP4Muo7r2AdVLoq2l+8YyezlKv6OLAJHCZSSXZHJkpngS+PUZMStP9gOw81 ZU1nK0jbBDPfIRuE+OziehGoF95/44QIvnZkorebaL8hV+eX/dptDfFyyRA/YO9bMqE1 DQBJ1KQlcbvr1Zc+67kiITHHb5haUtNu0dUl7ot3oiz7klldnuYaseMNAjC6scHZ7I6A mjliNHatagcSSFczz1veyxPD1MnwzyFPexsID8S1h4sZ/mrzKIFJd01cr4z0GooJJZi1 GzWCj+edGRMo5goZRWYk0pZ+zz/Qmxr04XSbD6VWvZRnxHrpFw6u5M/30suqn3kSx6gU 5Ymg== X-Gm-Message-State: AOAM530rXQ0guq2wWBrLetJJCe9B8U5U0/sP7YwQd0+jBRTb9795CtJa 6L2zOOdC5dmgrs2uWBevf15CUuCkZUNyofi3st6MFg== X-Google-Smtp-Source: ABdhPJx/iyiXGuXDn+GOqguExXZsb3ab/Q/oH21Oq8ykyLtEDUeHsHJxhR2K9nmUO8U7XGz9wS8dpYMRLmVJIOcZPLc= X-Received: by 2002:a0c:a162:: with SMTP id d89mr19139898qva.28.1600183238420; Tue, 15 Sep 2020 08:20:38 -0700 (PDT) MIME-Version: 1.0 References: <202008290429.07T4TrbH007764@repo.freebsd.org> <20200915141530.GM94807@kib.kiev.ua> <20200915142209.GA4145@raichu> <20200915143730.GB4145@raichu> In-Reply-To: <20200915143730.GB4145@raichu> From: Warner Losh Date: Tue, 15 Sep 2020 09:20:27 -0600 Message-ID: Subject: Re: svn commit: r364944 - head/sys/kern To: Mark Johnston Cc: Konstantin Belousov , Warner Losh , src-committers , svn-src-all , svn-src-head , Peter Holm X-Rspamd-Queue-Id: 4BrRkb4nwZz4FPn X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=boQitBEd; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::f29) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.86 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-0.89)[-0.889]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org,holm.cc]; NEURAL_HAM_LONG(-0.99)[-0.988]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-head@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; NEURAL_SPAM_SHORT(0.01)[0.014]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f29:from]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; MAILMAN_DEST(0.00)[svn-src-head] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 15:20:41 -0000 On Tue, Sep 15, 2020 at 8:37 AM Mark Johnston wrote: > On Tue, Sep 15, 2020 at 10:22:09AM -0400, Mark Johnston wrote: > > On Tue, Sep 15, 2020 at 05:15:30PM +0300, Konstantin Belousov wrote: > > > On Sat, Aug 29, 2020 at 04:29:53AM +0000, Warner Losh wrote: > > > > Author: imp > > > > Date: Sat Aug 29 04:29:53 2020 > > > > New Revision: 364944 > > > > URL: https://svnweb.freebsd.org/changeset/base/364944 > > > > > > > > Log: > > > > devctl: move to using a uma zone > > > > > > > > Convert the memory management of devctl. Rewrite if to make better > > > > use of memory. This eliminates several mallocs (5? worse case) > needed > > > > to send a message. It's now possible to always send a message, > though > > > > if things are really backed up the oldest message will be dropped > to > > > > free up space for the newest. > > > > > > > > Add a static bus_child_{location,pnpinfo}_sb to start migrating to > > > > sbuf instead of buffer + length. Use it in the new code. Other > code > > > > will be converted later (bus_child_*_str is only used inside of > > > > subr_bus.c, though implemented in ~100 places in the tree). > > > > > > > > Reviewed by: markj@ > > > > Differential Revision: https://reviews.freebsd.org/D26140 > > > > > > > > Modified: > > > > head/sys/kern/subr_bus.c > > > > > > > > > > > + } else { > > > > + /* dei can't be NULL -- we know we have at least one in > the zone */ > > > > + dei = uma_zalloc(devsoftc.zone, M_NOWAIT); > > > > + MPASS(dei != NULL); > > > This does not work. I believe you need to disable per-cpu cache for > the > > > zone, at least. But I am not sure it is enough. > > > > From the report we have: > > > > db:0:pho> show uma > > Zone Size Used Free Requests Sleeps Bucket > Total Mem XFree > > DEVCTL 1024 6415164 0 6416203 0 16 > 6569127936 0 > > > > so it looks like the primary problem is a leak. > > > > > https://people.freebsd.org/~pho/stress/log/kostik1314.txt > > devctl_queue() does not maintain the queue length bound, so if devd goes > away (due to an OOM kill in this case), we can end up with a large > backlog. This hack might be sufficient. > > diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c > index 19c056ab9974..91c57cdfae5b 100644 > --- a/sys/kern/subr_bus.c > +++ b/sys/kern/subr_bus.c > @@ -635,7 +635,14 @@ devctl_queue(struct dev_event_info *dei) > { > mtx_lock(&devsoftc.mtx); > STAILQ_INSERT_TAIL(&devsoftc.devq, dei, dei_link); > - devsoftc.queued++; > + if (devctl_queue_length != 0 && > + devctl_queue_length == devsoftc.queued) { > + dei = STAILQ_FIRST(&devsoftc.devq); > + STAILQ_REMOVE_HEAD(&devsoftc.devq, dei_link); > + uma_zfree(devsoftc.zone, dei); > + } else { > + devsoftc.queued++; > + } > cv_broadcast(&devsoftc.cv); > KNOTE_LOCKED(&devsoftc.sel.si_note, 0); > mtx_unlock(&devsoftc.mtx); > I've come to a similar conclusion... You can't queue w/o allocating... and that's where we're supposed to pop off the oldest, free it so we can allocate. So this isn't the right place for this fix, and the 'queued' name may be a bad name since it's from before. It's now supposed to be closer to 'allocated'. Warner