Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 17 Oct 2015 12:25:12 +0000
From:      "Cwalinski, Zygmunt" <zcwalinski@metroland.com>
To:        "freebsd-rc-request@freebsd.org" <freebsd-rc@freebsd.org>, "freebsd-rc@freebsd.org" <freebsd-rc@freebsd.org>
Subject:   Re: freebsd-rc Digest, Vol 470, Issue 1
Message-ID:  <20151017122510.30285913.76124.33808@metroland.com>

next in thread | raw e-mail | index | archive | help
freebsd-rc mailing list submissions to
freebsd-rc@freebsd.org

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.freebsd.org/mailman/listinfo/freebsd-rc
or, via email, send a message with subject or body 'help' to
freebsd-rc-request@freebsd.org

You can reach the person managing the list at
freebsd-rc-owner@freebsd.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of freebsd-rc digest..."


Today's Topics:

1. Re: rc(8) parallel tasks (Mark Felder)
2. Re: rc(8) parallel tasks (Cyril Vechera)
3. Re: rc(8) parallel tasks (Mark Martinec)


----------------------------------------------------------------------

Message: 1
Date: Fri, 16 Oct 2015 11:06:01 -0500
From: Mark Felder <feld@FreeBSD.org>
To: Cyril Vechera <cv@jet9.net>, dougb@FreeBSD.org,
freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org
Subject: Re: rc(8) parallel tasks
Message-ID:
<1445011561.1233840.412174489.688C5822@webmail.messagingengine.com>
Content-Type: text/plain



On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
> Hi there.
>=20
> We've got a small launcher script (~250 loc) for parallel services=20
> start/stop etc. It is used on our embedded systems and our users=20
> containers. And I've done a proof of concept for implanting it to the=20
> FreeBSD's standard /etc/rc for execution starting scripts in parallel.=20
> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly=
=20
> on eliminating jams for network or other long-latency resources waiting.
>=20
> The launcher is written in pure POSIX shell and uses FIFOs (named pipes)=
=20
> as a mutexes for synchronization. So it is embedded into /etc/rc and=20
> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it=20
> guarantees topological order (strict partial order) defined by=20
> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,=20
> mkfifo and a writeable file system. Due to last requirement, it can be=20
> run on the late stage or should be supplied by some kinf of writtable=20
> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder=20
> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change=20
> rc.d scripts
>=20
> It's not a full init replacement or a kind of services supervision tool.=
=20
> It only starts or invokes a group of scripts in parallel with resolving=20
> and assuring execution in dependencies order.
>=20
> Please take a look at the script and patch set for FreeBSD:
>=20
> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-in=
it
> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
>=20
>=20

Your first link is a 404, but this looks really nice!


--=20
Mark Felder
ports-secteam member
feld@FreeBSD.org


------------------------------

Message: 2
Date: Fri, 16 Oct 2015 21:37:27 +0300
From: Cyril Vechera <cv@jet9.net>
To: Mark Felder <feld@FreeBSD.org>
Cc: marck@rinet.ru, freebsd-hackers@freebsd.org,
freebsd-rc@freebsd.org
Subject: Re: rc(8) parallel tasks
Message-ID: <562143E7.3030104@jet9.net>
Content-Type: text/plain; charset=3DISO-8859-1; format=3Dflowed

On 10/16/2015 07:06 PM, Mark Felder wrote:
> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
>> Hi there.
>>
>> We've got a small launcher script (~250 loc) for parallel services
>> start/stop etc. It is used on our embedded systems and our users
>> containers. And I've done a proof of concept for implanting it to the
>> FreeBSD's standard /etc/rc for execution starting scripts in parallel.
>> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly
>> on eliminating jams for network or other long-latency resources waiting.
>>
>> The launcher is written in pure POSIX shell and uses FIFOs (named pipes)
>> as a mutexes for synchronization. So it is embedded into /etc/rc and
>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it
>> guarantees topological order (strict partial order) defined by
>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,
>> mkfifo and a writeable file system. Due to last requirement, it can be
>> run on the late stage or should be supplied by some kinf of writtable
>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder
>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change
>> rc.d scripts
>>
>> It's not a full init replacement or a kind of services supervision tool.
>> It only starts or invokes a group of scripts in parallel with resolving
>> and assuring execution in dependencies order.
>>
>> Please take a look at the script and patch set for FreeBSD:
>>
>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-i=
nit
>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd
>>
>>
> Your first link is a 404, but this looks really nice!

In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to=20
'jet9-multitask-flow' to avoid confusion with naming, because it's not=20
as it seems, from name, an init replacement, but just a parallel task=20
launcher. And now the actual repository URL is=20
https://github.com/cvss/jet9-multitask-flow

In this commit I've complete the FreeBSD compatibility. Now a script or=20
dependency name can include minuses `-` and dots `.` (the first stone I=20
stumbled over was ftp-proxy). And I've cleaned up the main script code=20
https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have=20
split it to more functions that can be redefined. So it's now easier to=20
rewrite dependency extraction for FreeBSD rc-scripts - current=20
implementation is too rough and takes three 'awk' runs for each=20
rc-script. The last is not only time loss, but as DMarck mentioned=20
before, using awk restricts parallel rc to be run only after FILESYSTEMS=20
stage is done. Maybe it would be better to add to the rcorder(8) some=20
new option to dump the gathered dependencies in tsort-compatible listing=20
and insert them directly to flow execution plan.

I've also added rc.conf variable `rc_parallel` to turn on and off=20
parallel execution. There's a risk to discover an incomplete dependency=20
annotations in some rc-scripts that earlier were masked by serial=20
execution. I've done some checks by enabling as much rc.conf variables=20
as possible to start more rc-scripts, and didn't found any error. But it=20
looks too good to be true and I'm afraid that it's just to poor testing.=20
I think if some ordering conflict will be found, it could be=20
worked-around with introducing a white-list for script names that must=20
be run only in sequentially.

So it remained first to check if it really works in different conditions.




--=20
Cyril Vechera



------------------------------

Message: 3
Date: Sat, 17 Oct 2015 00:14:01 +0200
From: Mark Martinec <Mark.Martinec+freebsd@ijs.si>
To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org
Cc: Cyril Vechera <cv@jet9.net>
Subject: Re: rc(8) parallel tasks
Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si>
Content-Type: text/plain; charset=3DUS-ASCII; format=3Dflowed

Tried a today's git checkout on one of our small virtual 10.2 boxes
with not many services. The installation is almost trivial and a
reboot went without any problems. Time for a reboot was shrunk by
about 10 seconds (which is understandable with a small number of
services). Very good, almost too good to be true for such fairly
small and non-intrusive piece of shell/awk scripting - thanks!

Mark


2015-10-16 20:37, Cyril Vechera wrote:
> On 10/16/2015 07:06 PM, Mark Felder wrote:
>> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote:
>>> Hi there.
>>>=20
>>> We've got a small launcher script (~250 loc) for parallel services
>>> start/stop etc. It is used on our embedded systems and our users
>>> containers. And I've done a proof of concept for implanting it to the
>>> FreeBSD's standard /etc/rc for execution starting scripts in=20
>>> parallel.
>>> It gave me a boot time reduction of rc part from 27 to 7 seconds,=20
>>> mostly
>>> on eliminating jams for network or other long-latency resources=20
>>> waiting.
>>>=20
>>> The launcher is written in pure POSIX shell and uses FIFOs (named=20
>>> pipes)
>>> as a mutexes for synchronization. So it is embedded into /etc/rc and
>>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it
>>> guarantees topological order (strict partial order) defined by
>>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,
>>> mkfifo and a writeable file system. Due to last requirement, it can=20
>>> be
>>> run on the late stage or should be supplied by some kinf of writtable
>>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder
>>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to=20
>>> change
>>> rc.d scripts
>>>=20
>>> It's not a full init replacement or a kind of services supervision=20
>>> tool.
>>> It only starts or invokes a group of scripts in parallel with=20
>>> resolving
>>> and assuring execution in dependencies order.
>>>=20
>>> Please take a look at the script and patch set for FreeBSD:
>>>=20
>>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-=
init
>>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebs=
d
>>>=20
>>>=20
>> Your first link is a 404, but this looks really nice!
>=20
> In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to
> 'jet9-multitask-flow' to avoid confusion with naming, because it's not
> as it seems, from name, an init replacement, but just a parallel task
> launcher. And now the actual repository URL is
> https://github.com/cvss/jet9-multitask-flow
>=20
> In this commit I've complete the FreeBSD compatibility. Now a script
> or dependency name can include minuses `-` and dots `.` (the first
> stone I stumbled over was ftp-proxy). And I've cleaned up the main
> script code
> https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and
> have split it to more functions that can be redefined. So it's now
> easier to rewrite dependency extraction for FreeBSD rc-scripts -
> current implementation is too rough and takes three 'awk' runs for
> each rc-script. The last is not only time loss, but as DMarck
> mentioned before, using awk restricts parallel rc to be run only after
> FILESYSTEMS stage is done. Maybe it would be better to add to the
> rcorder(8) some new option to dump the gathered dependencies in
> tsort-compatible listing and insert them directly to flow execution
> plan.
>=20
> I've also added rc.conf variable `rc_parallel` to turn on and off
> parallel execution. There's a risk to discover an incomplete
> dependency annotations in some rc-scripts that earlier were masked by
> serial execution. I've done some checks by enabling as much rc.conf
> variables as possible to start more rc-scripts, and didn't found any
> error. But it looks too good to be true and I'm afraid that it's just
> to poor testing. I think if some ordering conflict will be found, it
> could be worked-around with introducing a white-list for script names
> that must be run only in sequentially.
>=20
> So it remained first to check if it really works in different=20
> conditions.


------------------------------

Subject: Digest Footer

_______________________________________________
freebsd-rc@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-rc
To unsubscribe, send any mail to "freebsd-rc-unsubscribe@freebsd.org"

------------------------------

End of freebsd-rc Digest, Vol 470, Issue 1
******************************************



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151017122510.30285913.76124.33808>