Date: Mon, 13 Feb 2017 22:47:31 -0800 From: Cy Schubert <Cy.Schubert@komquats.com> To: Alan Somers <asomers@freebsd.org> Cc: Cy Schubert <Cy.Schubert@komquats.com>, scrappy@freebsd.org, Brian Somers <brian@freebsd.org>, freebsd-bugzilla@ayaken.net, Cy Schubert <cy@freebsd.org>, pkg@freebsd.org, =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= <des@freebsd.org> Subject: Re: Bug 217055 - Consolidate random sleeps in periodic scripts Message-ID: <201702140647.v1E6lVd8078946@slippy.cwsent.com> In-Reply-To: Message from Alan Somers <asomers@freebsd.org> of "Mon, 13 Feb 2017 16:00:31 -0700." <CAOtMX2jr6=-fmzni537JbzRv6f9YvsrHHPgnt5HnNLjLgjd5OQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <CAOtMX2jr6=-fmzni537JbzRv6f9YvsrHHPgnt5HnNLjLgjd5OQ@mail.gmail.c om> , Alan Somers writes: > On Mon, Feb 13, 2017 at 3:33 PM, Cy Schubert <Cy.Schubert@komquats.com> wrote > : > > In message <CAOtMX2jPWFBVxkPEhg3opFszmN9XmOvOvA_+DdeB9Rj6jg9ezw@mail.gmail. > c > > om> > > , Alan Somers writes: > >> On Mon, Feb 13, 2017 at 2:25 PM, Cy Schubert <Cy.Schubert@komquats.com> wr > ote > >> : > >> > In message <CAOtMX2jj_GCKjWW8CpapHutwH7OY00WnSWQS5VOuruv6i9Avqw@mail.gma > il. > >> c > >> > om> > >> > , Alan Somers writes: > >> >> On Mon, Feb 13, 2017 at 12:01 AM, Cy Schubert <Cy.Schubert@komquats.com > > w > >> rot > >> >> e: > >> >> > In message <CAOtMX2gJRuKKwwcHW5ZxTTZAm5Tmb7cVQ1SZEjwnuingYnO-Zg@mail. > gma > >> il. > >> >> c > >> >> > om> > >> >> > , Alan Somers writes: > >> >> >> I propose that we remove the various anti-congestion sleeps from > >> >> >> different periodic scripts, and add a single anti-congestion sleep t > o > >> >> >> the very beginning. Does this sound like a good idea to all of you? > >> >> >> > >> >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217055 > >> >> > > >> >> > I think the problem with the sleeps is simply the sleeps. My original > pl > >> an > >> >> > to put my sleep/fetch in the background was shot down by some who tho > ugh > >> t > >> >> > it wasn't simple enough. > >> >> > > >> >> > Secondly, we don't need sleeps every boot. Ntpd for example only need > s a > >> >> > sleep twice a year max to fetch a new leapfile so, to have a sleep ev > ery > >> >> > boot would be annoying. > >> >> > > >> >> > The best solution to replace sleeps would be to put a list of files:U > RLs > >> >> > into a queue to be fetched by fetcher script which would fetch only n > eed > >> ed > >> >> > files that boot (or in the case of ntp via periodic.conf twice a year > ). > >> >> > > >> >> > A single script with a queue of files to fetch with one anti-congesti > on > >> >> > sleep, preferably in the background. > >> >> > > >> >> > NTP, btw can (will) use the leapfile in /etc/ntp until a fresher copy > is > >> >> > fetched. > >> >> > > >> >> > Let's remove all fetching functions from the various rc scripts and q > ueu > >> e > >> >> > them up early in a fetcher rc script, preferably in the background if > at > >> >> > all possible. > >> >> > > >> >> > > >> >> > -- > >> >> > Cheers, > >> >> > Cy Schubert <Cy.Schubert@cschubert.com> > >> >> > FreeBSD UNIX: <cy@FreeBSD.org> Web: http://www.FreeBSD.org > >> >> > > >> >> > The need of the many outweighs the greed of the few. > >> >> > >> >> Unfortunately that won't work, Cy. Some scripts may need to > >> >> dynamically determine what files to fetch, in a way that we can't do > >> >> in a single separate fetcher script. Worse, some scripts, like > >> >> 300.statistics from sysutils/bsdstats, need to _post_ a URL, not get > >> >> one. > >> > > >> > Diverse requirements cannot be addressed by one knob. To assume that > >> > various applications all have the same sleep requirement won't work. > >> > >> Can you think any any periodic script whose sleep needs couldn't be > >> satisified by a single sleep at the beginning of the periodic run? I > >> can't. All sleeps I know of in /etc/periodic and > >> /usr/local/etc/periodic are for the purposes of reducing congestion > >> spikes on a server somewhere. The only way a single sleep could be > >> insufficient is if the random time interval is too small. > > > > Yes. ntpd. It doesn't need a sleep every time periodic is run. It only > > needs a sleep once during the 30 day period prior to leap-second file > > expiry. To impose a sleep for ntpd every time periodic is run is a waste of > > time. > > > >> > >> > > >> > I suppose we could have an optional single sleep script but we can't > >> > summarily remove all sleeps and assume all rc and periodic scripts sleep > >> > for some, one or possibly no applications requiring a sleep at any given > >> > time. > >> > >> What? I can't make sense of that sentence. > > > > What I'm saying is if you want a global sleep, have it as an option, not a > > replacement that forces a sleep when none is needed. > > > >> > >> > We can have a general sleep but removing the option of others would > >> > be counter productive. It doesn't make sense to have an arbitrary sleep > >> > just in case a subsequent script might need it. > >> > >> Nothing in /etc/periodic needs to be run at a precise time, so adding > >> a sleep won't hurt anything. And if the sleep is configurable, a > >> sysadmin can always disable it. Also, from an anticongestion > >> standpoint it's objectively less good to chain multiple sleeps > >> together instead of using a single longer sleep. The reason is > >> because when you add several uniformly distributed random variables, > >> the result approaches a normal distribution with a peak in the middle. > >> But for anticongestion purposes, a uniform distribution is really what > >> you want. > >> > >> > If we have to, let's either > >> > reduce the length of the sleeps or put better yet background them. > >> > >> I don't like the idea of backgrounding parts of the periodic scripts, > >> for three reasons. One, it's complicated. Two, it prevents > >> periodic(8) from sending a single status email. Three, periodic(8) > >> might start the next day's run before the previous day's is complete. > > > > It is more complicated, yes. However a sleep delays sending the email and > > agreed you don't want it running the next day. > > > >> > >> > > >> > What's motivating this? Server? Laptop? > >> > >> Servers mostly. A confounding issue is Bug 210188 - periodic daily > >> sleeps even when invoked from a terminal . When I run periodic by > >> hand, it still sleeps. It would be easier to fix that bug if only one > >> sleep were involved instead of several. > > > > That can be solved by, > > if tty >/dev/null 2>&1; then > > SLEEP_TIME=0 > > else > > SLEEP_TIME=some algorithm to randomize sleep > > fi > > > > This could be put in rc.subr or some other common place and sourced by > > scripts that need it. Of course this doesn't address ports that issue > > sleep. Nor would a common 000 sleep script either. Future ports and porters > > will continue to put random sleeps in ports as well. > > That's a good idea. Currently, no periodic script includes > /etc/rc.subr, but we can (and already do) define functions in > /etc/defaults/periodic.conf. I can put an anticongest function in > there. It can use a /var/run file in conjunction with an environment > variable set by periodic(8) to ensure that the sleep happens no more > than once for each invocation of periodic(8). > > Also, I don't think ports is going to be a big problem. Currently I > only know of two ports whose periodic scripts include a sleep. One is > bsdstat, whose periodic script is entirely contained within the ports > tree. The other is pkg, which is also controlled by the FreeBSD > project. So it should be straightforward to convert them. Hi Alan, Looking at ntp, it backgrounds itself: (sleep $(jot -r 1 0 3600); service ntpd onefetch) & To facilitate debugging, I can commit the following, if you don't mind. Index: periodic/daily/480.leapfile-ntpd =================================================================== --- periodic/daily/480.leapfile-ntpd (revision 313710) +++ periodic/daily/480.leapfile-ntpd (working copy) @@ -13,6 +13,7 @@ case "$daily_ntpd_leapfile_enable" in [Yy][Ee][Ss]) + tty >/dev/null 2>&1 && daily_ntpd_avoid_congestion=NO case "$daily_ntpd_avoid_congestion" in [Yy][Ee][Ss]) # Avoid dogpiling -- Cheers, Cy Schubert <Cy.Schubert@cschubert.com> FreeBSD UNIX: <cy@FreeBSD.org> Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201702140647.v1E6lVd8078946>