Date: Sat, 17 Oct 2015 12:25:12 +0000 From: "Cwalinski, Zygmunt" <zcwalinski@metroland.com> To: "freebsd-rc-request@freebsd.org" <freebsd-rc@freebsd.org>, "freebsd-rc@freebsd.org" <freebsd-rc@freebsd.org> Subject: Re: freebsd-rc Digest, Vol 470, Issue 1 Message-ID: <20151017122510.30285913.76124.33808@metroland.com>
next in thread | raw e-mail | index | archive | help
freebsd-rc mailing list submissions to freebsd-rc@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.freebsd.org/mailman/listinfo/freebsd-rc or, via email, send a message with subject or body 'help' to freebsd-rc-request@freebsd.org You can reach the person managing the list at freebsd-rc-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-rc digest..." Today's Topics: 1. Re: rc(8) parallel tasks (Mark Felder) 2. Re: rc(8) parallel tasks (Cyril Vechera) 3. Re: rc(8) parallel tasks (Mark Martinec) ---------------------------------------------------------------------- Message: 1 Date: Fri, 16 Oct 2015 11:06:01 -0500 From: Mark Felder <feld@FreeBSD.org> To: Cyril Vechera <cv@jet9.net>, dougb@FreeBSD.org, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> Content-Type: text/plain On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: > Hi there. >=20 > We've got a small launcher script (~250 loc) for parallel services=20 > start/stop etc. It is used on our embedded systems and our users=20 > containers. And I've done a proof of concept for implanting it to the=20 > FreeBSD's standard /etc/rc for execution starting scripts in parallel.=20 > It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly= =20 > on eliminating jams for network or other long-latency resources waiting. >=20 > The launcher is written in pure POSIX shell and uses FIFOs (named pipes)= =20 > as a mutexes for synchronization. So it is embedded into /etc/rc and=20 > /etc/rc.d preserving rc.subr preloading. As a primary requirement, it=20 > guarantees topological order (strict partial order) defined by=20 > dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,=20 > mkfifo and a writeable file system. Due to last requirement, it can be=20 > run on the late stage or should be supplied by some kinf of writtable=20 > fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder=20 > annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change=20 > rc.d scripts >=20 > It's not a full init replacement or a kind of services supervision tool.= =20 > It only starts or invokes a group of scripts in parallel with resolving=20 > and assuring execution in dependencies order. >=20 > Please take a look at the script and patch set for FreeBSD: >=20 > https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-in= it > https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >=20 >=20 Your first link is a 404, but this looks really nice! --=20 Mark Felder ports-secteam member feld@FreeBSD.org ------------------------------ Message: 2 Date: Fri, 16 Oct 2015 21:37:27 +0300 From: Cyril Vechera <cv@jet9.net> To: Mark Felder <feld@FreeBSD.org> Cc: marck@rinet.ru, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <562143E7.3030104@jet9.net> Content-Type: text/plain; charset=3DISO-8859-1; format=3Dflowed On 10/16/2015 07:06 PM, Mark Felder wrote: > On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >> Hi there. >> >> We've got a small launcher script (~250 loc) for parallel services >> start/stop etc. It is used on our embedded systems and our users >> containers. And I've done a proof of concept for implanting it to the >> FreeBSD's standard /etc/rc for execution starting scripts in parallel. >> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly >> on eliminating jams for network or other long-latency resources waiting. >> >> The launcher is written in pure POSIX shell and uses FIFOs (named pipes) >> as a mutexes for synchronization. So it is embedded into /etc/rc and >> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >> guarantees topological order (strict partial order) defined by >> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >> mkfifo and a writeable file system. Due to last requirement, it can be >> run on the late stage or should be supplied by some kinf of writtable >> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change >> rc.d scripts >> >> It's not a full init replacement or a kind of services supervision tool. >> It only starts or invokes a group of scripts in parallel with resolving >> and assuring execution in dependencies order. >> >> Please take a look at the script and patch set for FreeBSD: >> >> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-i= nit >> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >> >> > Your first link is a 404, but this looks really nice! In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to=20 'jet9-multitask-flow' to avoid confusion with naming, because it's not=20 as it seems, from name, an init replacement, but just a parallel task=20 launcher. And now the actual repository URL is=20 https://github.com/cvss/jet9-multitask-flow In this commit I've complete the FreeBSD compatibility. Now a script or=20 dependency name can include minuses `-` and dots `.` (the first stone I=20 stumbled over was ftp-proxy). And I've cleaned up the main script code=20 https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have=20 split it to more functions that can be redefined. So it's now easier to=20 rewrite dependency extraction for FreeBSD rc-scripts - current=20 implementation is too rough and takes three 'awk' runs for each=20 rc-script. The last is not only time loss, but as DMarck mentioned=20 before, using awk restricts parallel rc to be run only after FILESYSTEMS=20 stage is done. Maybe it would be better to add to the rcorder(8) some=20 new option to dump the gathered dependencies in tsort-compatible listing=20 and insert them directly to flow execution plan. I've also added rc.conf variable `rc_parallel` to turn on and off=20 parallel execution. There's a risk to discover an incomplete dependency=20 annotations in some rc-scripts that earlier were masked by serial=20 execution. I've done some checks by enabling as much rc.conf variables=20 as possible to start more rc-scripts, and didn't found any error. But it=20 looks too good to be true and I'm afraid that it's just to poor testing.=20 I think if some ordering conflict will be found, it could be=20 worked-around with introducing a white-list for script names that must=20 be run only in sequentially. So it remained first to check if it really works in different conditions. --=20 Cyril Vechera ------------------------------ Message: 3 Date: Sat, 17 Oct 2015 00:14:01 +0200 From: Mark Martinec <Mark.Martinec+freebsd@ijs.si> To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Cc: Cyril Vechera <cv@jet9.net> Subject: Re: rc(8) parallel tasks Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si> Content-Type: text/plain; charset=3DUS-ASCII; format=3Dflowed Tried a today's git checkout on one of our small virtual 10.2 boxes with not many services. The installation is almost trivial and a reboot went without any problems. Time for a reboot was shrunk by about 10 seconds (which is understandable with a small number of services). Very good, almost too good to be true for such fairly small and non-intrusive piece of shell/awk scripting - thanks! Mark 2015-10-16 20:37, Cyril Vechera wrote: > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>>=20 >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to the >>> FreeBSD's standard /etc/rc for execution starting scripts in=20 >>> parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds,=20 >>> mostly >>> on eliminating jams for network or other long-latency resources=20 >>> waiting. >>>=20 >>> The launcher is written in pure POSIX shell and uses FIFOs (named=20 >>> pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can=20 >>> be >>> run on the late stage or should be supplied by some kinf of writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to=20 >>> change >>> rc.d scripts >>>=20 >>> It's not a full init replacement or a kind of services supervision=20 >>> tool. >>> It only starts or invokes a group of scripts in parallel with=20 >>> resolving >>> and assuring execution in dependencies order. >>>=20 >>> Please take a look at the script and patch set for FreeBSD: >>>=20 >>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-= init >>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebs= d >>>=20 >>>=20 >> Your first link is a 404, but this looks really nice! >=20 > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to > 'jet9-multitask-flow' to avoid confusion with naming, because it's not > as it seems, from name, an init replacement, but just a parallel task > launcher. And now the actual repository URL is > https://github.com/cvss/jet9-multitask-flow >=20 > In this commit I've complete the FreeBSD compatibility. Now a script > or dependency name can include minuses `-` and dots `.` (the first > stone I stumbled over was ftp-proxy). And I've cleaned up the main > script code > https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and > have split it to more functions that can be redefined. So it's now > easier to rewrite dependency extraction for FreeBSD rc-scripts - > current implementation is too rough and takes three 'awk' runs for > each rc-script. The last is not only time loss, but as DMarck > mentioned before, using awk restricts parallel rc to be run only after > FILESYSTEMS stage is done. Maybe it would be better to add to the > rcorder(8) some new option to dump the gathered dependencies in > tsort-compatible listing and insert them directly to flow execution > plan. >=20 > I've also added rc.conf variable `rc_parallel` to turn on and off > parallel execution. There's a risk to discover an incomplete > dependency annotations in some rc-scripts that earlier were masked by > serial execution. I've done some checks by enabling as much rc.conf > variables as possible to start more rc-scripts, and didn't found any > error. But it looks too good to be true and I'm afraid that it's just > to poor testing. I think if some ordering conflict will be found, it > could be worked-around with introducing a white-list for script names > that must be run only in sequentially. >=20 > So it remained first to check if it really works in different=20 > conditions. ------------------------------ Subject: Digest Footer _______________________________________________ freebsd-rc@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-rc To unsubscribe, send any mail to "freebsd-rc-unsubscribe@freebsd.org" ------------------------------ End of freebsd-rc Digest, Vol 470, Issue 1 ******************************************
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151017122510.30285913.76124.33808>