Date: Mon, 13 Feb 2017 14:33:41 -0800 From: Cy Schubert <Cy.Schubert@komquats.com> To: Alan Somers <asomers@freebsd.org> Cc: Cy Schubert <Cy.Schubert@komquats.com>, scrappy@freebsd.org, Brian Somers <brian@freebsd.org>, freebsd-bugzilla@ayaken.net, Cy Schubert <cy@freebsd.org>, pkg@freebsd.org, des@freebsd.org Subject: Re: Bug 217055 - Consolidate random sleeps in periodic scripts Message-ID: <201702132233.v1DMXf4P092101@slippy.cwsent.com> In-Reply-To: Message from Alan Somers <asomers@freebsd.org> of "Mon, 13 Feb 2017 15:07:17 -0700." <CAOtMX2jPWFBVxkPEhg3opFszmN9XmOvOvA_%2BDdeB9Rj6jg9ezw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <CAOtMX2jPWFBVxkPEhg3opFszmN9XmOvOvA_+DdeB9Rj6jg9ezw@mail.gmail.c om> , Alan Somers writes: > On Mon, Feb 13, 2017 at 2:25 PM, Cy Schubert <Cy.Schubert@komquats.com> wrote > : > > In message <CAOtMX2jj_GCKjWW8CpapHutwH7OY00WnSWQS5VOuruv6i9Avqw@mail.gmail. > c > > om> > > , Alan Somers writes: > >> On Mon, Feb 13, 2017 at 12:01 AM, Cy Schubert <Cy.Schubert@komquats.com> w > rot > >> e: > >> > In message <CAOtMX2gJRuKKwwcHW5ZxTTZAm5Tmb7cVQ1SZEjwnuingYnO-Zg@mail.gma > il. > >> c > >> > om> > >> > , Alan Somers writes: > >> >> I propose that we remove the various anti-congestion sleeps from > >> >> different periodic scripts, and add a single anti-congestion sleep to > >> >> the very beginning. Does this sound like a good idea to all of you? > >> >> > >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217055 > >> > > >> > I think the problem with the sleeps is simply the sleeps. My original pl > an > >> > to put my sleep/fetch in the background was shot down by some who though > t > >> > it wasn't simple enough. > >> > > >> > Secondly, we don't need sleeps every boot. Ntpd for example only needs a > >> > sleep twice a year max to fetch a new leapfile so, to have a sleep every > >> > boot would be annoying. > >> > > >> > The best solution to replace sleeps would be to put a list of files:URLs > >> > into a queue to be fetched by fetcher script which would fetch only need > ed > >> > files that boot (or in the case of ntp via periodic.conf twice a year). > >> > > >> > A single script with a queue of files to fetch with one anti-congestion > >> > sleep, preferably in the background. > >> > > >> > NTP, btw can (will) use the leapfile in /etc/ntp until a fresher copy is > >> > fetched. > >> > > >> > Let's remove all fetching functions from the various rc scripts and queu > e > >> > them up early in a fetcher rc script, preferably in the background if at > >> > all possible. > >> > > >> > > >> > -- > >> > Cheers, > >> > Cy Schubert <Cy.Schubert@cschubert.com> > >> > FreeBSD UNIX: <cy@FreeBSD.org> Web: http://www.FreeBSD.org > >> > > >> > The need of the many outweighs the greed of the few. > >> > >> Unfortunately that won't work, Cy. Some scripts may need to > >> dynamically determine what files to fetch, in a way that we can't do > >> in a single separate fetcher script. Worse, some scripts, like > >> 300.statistics from sysutils/bsdstats, need to _post_ a URL, not get > >> one. > > > > Diverse requirements cannot be addressed by one knob. To assume that > > various applications all have the same sleep requirement won't work. > > Can you think any any periodic script whose sleep needs couldn't be > satisified by a single sleep at the beginning of the periodic run? I > can't. All sleeps I know of in /etc/periodic and > /usr/local/etc/periodic are for the purposes of reducing congestion > spikes on a server somewhere. The only way a single sleep could be > insufficient is if the random time interval is too small. Yes. ntpd. It doesn't need a sleep every time periodic is run. It only needs a sleep once during the 30 day period prior to leap-second file expiry. To impose a sleep for ntpd every time periodic is run is a waste of time. > > > > > I suppose we could have an optional single sleep script but we can't > > summarily remove all sleeps and assume all rc and periodic scripts sleep > > for some, one or possibly no applications requiring a sleep at any given > > time. > > What? I can't make sense of that sentence. What I'm saying is if you want a global sleep, have it as an option, not a replacement that forces a sleep when none is needed. > > > We can have a general sleep but removing the option of others would > > be counter productive. It doesn't make sense to have an arbitrary sleep > > just in case a subsequent script might need it. > > Nothing in /etc/periodic needs to be run at a precise time, so adding > a sleep won't hurt anything. And if the sleep is configurable, a > sysadmin can always disable it. Also, from an anticongestion > standpoint it's objectively less good to chain multiple sleeps > together instead of using a single longer sleep. The reason is > because when you add several uniformly distributed random variables, > the result approaches a normal distribution with a peak in the middle. > But for anticongestion purposes, a uniform distribution is really what > you want. > > > If we have to, let's either > > reduce the length of the sleeps or put better yet background them. > > I don't like the idea of backgrounding parts of the periodic scripts, > for three reasons. One, it's complicated. Two, it prevents > periodic(8) from sending a single status email. Three, periodic(8) > might start the next day's run before the previous day's is complete. It is more complicated, yes. However a sleep delays sending the email and agreed you don't want it running the next day. > > > > > What's motivating this? Server? Laptop? > > Servers mostly. A confounding issue is Bug 210188 - periodic daily > sleeps even when invoked from a terminal . When I run periodic by > hand, it still sleeps. It would be easier to fix that bug if only one > sleep were involved instead of several. That can be solved by, if tty >/dev/null 2>&1; then SLEEP_TIME=0 else SLEEP_TIME=some algorithm to randomize sleep fi This could be put in rc.subr or some other common place and sourced by scripts that need it. Of course this doesn't address ports that issue sleep. Nor would a common 000 sleep script either. Future ports and porters will continue to put random sleeps in ports as well. The above avoids imposing a sleep on every periodic script whether it wants it or not just in case a periodic script MIGHT just need a sleep. I've added des to the cc list. He will be interested in this discussion. -- Cheers, Cy Schubert <Cy.Schubert@cschubert.com> FreeBSD UNIX: <cy@FreeBSD.org> Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201702132233.v1DMXf4P092101>