From owner-freebsd-arch@freebsd.org Thu Oct 8 16:18:56 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4713B9D2B04 for ; Thu, 8 Oct 2015 16:18:56 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-f50.google.com (mail-qg0-f50.google.com [209.85.192.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0BD108D9 for ; Thu, 8 Oct 2015 16:18:55 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by qgt47 with SMTP id 47so46217520qgt.2 for ; Thu, 08 Oct 2015 09:18:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=9jt3h8ygyKLpASM8q3xbYoqMOesQg2a2gKNXuhupiss=; b=FHSLB/g8LpvkoiM2bSuqZbvIhmdTFsxF4JPpHOKEPZCLiBEwzOhk9aApyRfixqIv9O oVh3pHV4bcK1EP1Vl4O3mLr5mhtwxkoivl7yQMlCWJMdpDL+R1sGNliiEJSx3QCM2+S0 Wb4yEm/noYxsv8jSppBLsjBzMaFCzIaf1gjvXyNPr/i5nA0rKD7ndxvkdfRNNlU8biJx UjcY0/RB1ZMDAOIhc5EZL5e5TG/chGNZUCFA3nQAE06d9Vqpw3K8ihT2UoWwwcI6er4y iO9vnfmsHaR1+Zm97zn9q3fM0n3I610Pv55LKO0s+e29p1sOJXhNLKBelpASUuO9xxMX 6y+A== X-Gm-Message-State: ALoCoQm4j27cS6GUX2pFA1CQDpoN2KdUVmEFTGCWCUm5cPIYiGdv+CbnvI7h1+sGIX+mf0feoZjC MIME-Version: 1.0 X-Received: by 10.140.232.210 with SMTP id d201mr9858780qhc.94.1444320782834; Thu, 08 Oct 2015 09:13:02 -0700 (PDT) Sender: wlosh@bsdimp.com Received: by 10.140.80.167 with HTTP; Thu, 8 Oct 2015 09:13:02 -0700 (PDT) X-Originating-IP: [50.253.99.174] In-Reply-To: <5615886F.3060601@FreeBSD.org> References: <5615886F.3060601@FreeBSD.org> Date: Thu, 8 Oct 2015 10:13:02 -0600 X-Google-Sender-Auth: SxBNzAZeLrTrYBCfk1fNUHGv8Rs Message-ID: Subject: Re: RFC: Automatically Reloading /etc/resolv.conf From: Warner Losh To: Eric van Gyzen Cc: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Oct 2015 16:18:56 -0000 On Wed, Oct 7, 2015 at 3:02 PM, Eric van Gyzen wrote: > I would like to change the libc resolver to automatically reload > /etc/resolv.conf when the latter changes. I would like to hear opinions > about the implementation. Broadly, I see two approaches. > > == "stat" == > > When loading the file, record the mod time. Before each query, stat() > the file to see if it has changed. > > Advantage: It uses no extra persistently allocated objects. > > Disadvantage: It incurs a stat() on every query. I don't see this as a > major disadvantage, since the resolver already does a lot of work on > every query. (For example, it creates and destroys a kqueue and a socket.) > > OpenBSD uses this approach. It also uses clock_gettime(CLOCK_MONOTONIC) > to rate-limit the stat() calls to one per several seconds. > Make sure you rate limit it. Calling stat at a high rate from multiple CPUs exposes poor locking scaleability and eats a lot of CPU. We had some setting wrong on NGINX at work that lead to large NGINX latency spikes because this contention was blocking the worker threads. Correcting the settings to cache the results of stat made this problem disappear. > == "kqueue" == > > When loading the file, open a kqueue and register for the appropriate > events. Before each query, check for kevents. > > Advantage: The per-query overhead is fairly small. > > Disadvantage: This would persistently allocate an open file and a > kqueue for every thread that ever uses the resolver, for the life of the > thread. This seems fairly expensive. > Why does this follow? Can't you have a global one for the process? > NetBSD uses this approach. It mitigates most of the space-cost by using > a shared pool of res_state objects, instead of one per thread [that uses > the resolver]. On each query, a thread allocates/borrows a res_state > from the pool, uses it, and returns it. So, the number of objects is > only the high water mark of the number of threads _concurrently_ issuing > resolver queries.n > What's the locking scaleability here when resolv.conf changes? There are probably several variations on each theme, of course. I would > appreciate your thoughts on these approaches and others I missed, as > well as variations and details. > > FYI, I'm leaning toward the "stat" approach. > It sounds simpler to implement, but likely a higher overhead than the kqueue approach. resolv.conf changes on an time scale measured in minutes or hours for the mobile user, and on the scale of years for servers. Polling on the scale of seconds (at least two orders of magnitude faster than the rate of change) seems like a lot of extra work over the life of resolv.conf. Warner