From owner-freebsd-rc@freebsd.org Sat Oct 17 12:25:19 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41BC9A1759E for ; Sat, 17 Oct 2015 12:25:19 +0000 (UTC) (envelope-from zcwalinski@metroland.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0070.outbound.protection.outlook.com [157.56.111.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF2E1107F for ; Sat, 17 Oct 2015 12:25:17 +0000 (UTC) (envelope-from zcwalinski@metroland.com) Received: from CY1PR08MB1415.namprd08.prod.outlook.com (10.161.172.142) by CY1PR08MB1989.namprd08.prod.outlook.com (10.164.222.21) with Microsoft SMTP Server (TLS) id 15.1.300.14; Sat, 17 Oct 2015 12:25:15 +0000 Received: from CY1PR08MB1413.namprd08.prod.outlook.com (10.161.172.140) by CY1PR08MB1415.namprd08.prod.outlook.com (10.161.172.142) with Microsoft SMTP Server (TLS) id 15.1.300.14; Sat, 17 Oct 2015 12:25:12 +0000 Received: from CY1PR08MB1413.namprd08.prod.outlook.com ([10.161.172.140]) by CY1PR08MB1413.namprd08.prod.outlook.com ([10.161.172.140]) with mapi id 15.01.0300.010; Sat, 17 Oct 2015 12:25:12 +0000 From: "Cwalinski, Zygmunt" To: "freebsd-rc-request@freebsd.org" , "freebsd-rc@freebsd.org" Subject: Re: freebsd-rc Digest, Vol 470, Issue 1 Thread-Topic: freebsd-rc Digest, Vol 470, Issue 1 Thread-Index: AQHRCNbew6EpJhUqrUuXNABeON5t0A== Date: Sat, 17 Oct 2015 12:25:12 +0000 Message-ID: <20151017122510.30285913.76124.33808@metroland.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-client-id: 33808 authentication-results: spf=none (sender IP is ) smtp.mailfrom=zcwalinski@metroland.com; x-originating-ip: [69.165.175.208] x-microsoft-exchange-diagnostics: 1; CY1PR08MB1415; 5:C5P0uDFuHIvixuH9UAnVnBghLnM/VvdWERsMd6ePvNZaJ4j3WRLwGAqkUc+mukZuR5Qca2BgHeuLvr+Lx2NNgH6w8LIXsLM9VQsj6E905EsSl5FrxReYfLXdHpn5hztr+sJGFCqV3anroyeFJv0LbQ==; 24:yoX083EwkBY2HpcICdZyyPcH5kBIC53dplclw0nxTgVPDWo4BOCj06IQIcoh+NBfLicHFIJZSUHiz5mjnwWOoI5w1JTr0qGzJbyWiaSX5Sk=; 20:3JdLUyDdsJ9uz+NUuZGHf+JSbCSWo6sHU5HYyZr+eVWq2ozpssiZxJDTOJ3vxOBbV1qhJ8Iddg0Hc1cy4oemsA== x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(42134001)(42139001); SRVR:CY1PR08MB1415; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001); SRVR:CY1PR08MB1415; BCL:0; PCL:0; RULEID:; SRVR:CY1PR08MB1415; x-forefront-prvs: 07326CFBC4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377424004)(24454002)(479174004)(27574002)(18543002)(5403001)(199003)(10533003)(189002)(377454003)(66066001)(106356001)(50986999)(77096005)(122556002)(101416001)(64706001)(102836002)(5008740100001)(36756003)(103116003)(92566002)(81156007)(97736004)(40100003)(566704002)(11100500001)(4001150100001)(54356999)(5001770100001)(5002640100001)(99286002)(86362001)(450100001)(15975445007)(10400500002)(2900100001)(5007970100001)(100306002)(189998001)(105586002)(19580405001)(106116001)(5004730100002)(87936001)(5001960100002)(19580395003)(46102003)(107886002)(2501003)(491001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR08MB1415; H:CY1PR08MB1413.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: metroland.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <3A3AE62B261D8D488BA4BA0B468C4A47@torstarcorp.onmicrosoft.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Oct 2015 12:25:12.4124 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: e4142d33-522c-4393-ac12-a97843392e58 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR08MB1415 X-Microsoft-Exchange-Diagnostics: 1; CY1PR08MB1989; 2:vBA9MjymWOGxjKIdQ2Za7zED32h3al3f75Hu2zIZPgrMxvF5D3riQdQtPb+cCtTUTc7twvEI19Zme4/ogv54UV5kBzgkdjgL55RCY1xkBohJCWDBHhUEqRmZmxVMIuCdic0H1PTmp9MzrE1d9RvqcIkJvGdeCyhHzk0yOMKIbo8=; 23:ZH60WuQ9BisyIlmZL7AZ50N/6TpYyujwlntajrjNgKuxd6UrcOGlYWXneiDX4okQZzXaA7qs/xUPVBLOdxtasO19Su1ciaeHchRCOniL20lrkKCLymmScVO6S9x6Y7SVoADtT/mE2AXNckuJz+8urKN999NDmf3KvTYZ3jgy3OtTqhCH3Oiik4SV7BcATzJy X-OriginatorOrg: metroland.com X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 12:25:19 -0000 freebsd-rc mailing list submissions to freebsd-rc@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.freebsd.org/mailman/listinfo/freebsd-rc or, via email, send a message with subject or body 'help' to freebsd-rc-request@freebsd.org You can reach the person managing the list at freebsd-rc-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-rc digest..." Today's Topics: 1. Re: rc(8) parallel tasks (Mark Felder) 2. Re: rc(8) parallel tasks (Cyril Vechera) 3. Re: rc(8) parallel tasks (Mark Martinec) ---------------------------------------------------------------------- Message: 1 Date: Fri, 16 Oct 2015 11:06:01 -0500 From: Mark Felder To: Cyril Vechera , dougb@FreeBSD.org, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> Content-Type: text/plain On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: > Hi there. >=20 > We've got a small launcher script (~250 loc) for parallel services=20 > start/stop etc. It is used on our embedded systems and our users=20 > containers. And I've done a proof of concept for implanting it to the=20 > FreeBSD's standard /etc/rc for execution starting scripts in parallel.=20 > It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly= =20 > on eliminating jams for network or other long-latency resources waiting. >=20 > The launcher is written in pure POSIX shell and uses FIFOs (named pipes)= =20 > as a mutexes for synchronization. So it is embedded into /etc/rc and=20 > /etc/rc.d preserving rc.subr preloading. As a primary requirement, it=20 > guarantees topological order (strict partial order) defined by=20 > dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,=20 > mkfifo and a writeable file system. Due to last requirement, it can be=20 > run on the late stage or should be supplied by some kinf of writtable=20 > fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder=20 > annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change=20 > rc.d scripts >=20 > It's not a full init replacement or a kind of services supervision tool.= =20 > It only starts or invokes a group of scripts in parallel with resolving=20 > and assuring execution in dependencies order. >=20 > Please take a look at the script and patch set for FreeBSD: >=20 > https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-in= it > https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >=20 >=20 Your first link is a 404, but this looks really nice! --=20 Mark Felder ports-secteam member feld@FreeBSD.org ------------------------------ Message: 2 Date: Fri, 16 Oct 2015 21:37:27 +0300 From: Cyril Vechera To: Mark Felder Cc: marck@rinet.ru, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <562143E7.3030104@jet9.net> Content-Type: text/plain; charset=3DISO-8859-1; format=3Dflowed On 10/16/2015 07:06 PM, Mark Felder wrote: > On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >> Hi there. >> >> We've got a small launcher script (~250 loc) for parallel services >> start/stop etc. It is used on our embedded systems and our users >> containers. And I've done a proof of concept for implanting it to the >> FreeBSD's standard /etc/rc for execution starting scripts in parallel. >> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly >> on eliminating jams for network or other long-latency resources waiting. >> >> The launcher is written in pure POSIX shell and uses FIFOs (named pipes) >> as a mutexes for synchronization. So it is embedded into /etc/rc and >> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >> guarantees topological order (strict partial order) defined by >> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >> mkfifo and a writeable file system. Due to last requirement, it can be >> run on the late stage or should be supplied by some kinf of writtable >> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change >> rc.d scripts >> >> It's not a full init replacement or a kind of services supervision tool. >> It only starts or invokes a group of scripts in parallel with resolving >> and assuring execution in dependencies order. >> >> Please take a look at the script and patch set for FreeBSD: >> >> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-i= nit >> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >> >> > Your first link is a 404, but this looks really nice! In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to=20 'jet9-multitask-flow' to avoid confusion with naming, because it's not=20 as it seems, from name, an init replacement, but just a parallel task=20 launcher. And now the actual repository URL is=20 https://github.com/cvss/jet9-multitask-flow In this commit I've complete the FreeBSD compatibility. Now a script or=20 dependency name can include minuses `-` and dots `.` (the first stone I=20 stumbled over was ftp-proxy). And I've cleaned up the main script code=20 https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have=20 split it to more functions that can be redefined. So it's now easier to=20 rewrite dependency extraction for FreeBSD rc-scripts - current=20 implementation is too rough and takes three 'awk' runs for each=20 rc-script. The last is not only time loss, but as DMarck mentioned=20 before, using awk restricts parallel rc to be run only after FILESYSTEMS=20 stage is done. Maybe it would be better to add to the rcorder(8) some=20 new option to dump the gathered dependencies in tsort-compatible listing=20 and insert them directly to flow execution plan. I've also added rc.conf variable `rc_parallel` to turn on and off=20 parallel execution. There's a risk to discover an incomplete dependency=20 annotations in some rc-scripts that earlier were masked by serial=20 execution. I've done some checks by enabling as much rc.conf variables=20 as possible to start more rc-scripts, and didn't found any error. But it=20 looks too good to be true and I'm afraid that it's just to poor testing.=20 I think if some ordering conflict will be found, it could be=20 worked-around with introducing a white-list for script names that must=20 be run only in sequentially. So it remained first to check if it really works in different conditions. --=20 Cyril Vechera ------------------------------ Message: 3 Date: Sat, 17 Oct 2015 00:14:01 +0200 From: Mark Martinec To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Cc: Cyril Vechera Subject: Re: rc(8) parallel tasks Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si> Content-Type: text/plain; charset=3DUS-ASCII; format=3Dflowed Tried a today's git checkout on one of our small virtual 10.2 boxes with not many services. The installation is almost trivial and a reboot went without any problems. Time for a reboot was shrunk by about 10 seconds (which is understandable with a small number of services). Very good, almost too good to be true for such fairly small and non-intrusive piece of shell/awk scripting - thanks! Mark 2015-10-16 20:37, Cyril Vechera wrote: > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>>=20 >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to the >>> FreeBSD's standard /etc/rc for execution starting scripts in=20 >>> parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds,=20 >>> mostly >>> on eliminating jams for network or other long-latency resources=20 >>> waiting. >>>=20 >>> The launcher is written in pure POSIX shell and uses FIFOs (named=20 >>> pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can=20 >>> be >>> run on the late stage or should be supplied by some kinf of writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to=20 >>> change >>> rc.d scripts >>>=20 >>> It's not a full init replacement or a kind of services supervision=20 >>> tool. >>> It only starts or invokes a group of scripts in parallel with=20 >>> resolving >>> and assuring execution in dependencies order. >>>=20 >>> Please take a look at the script and patch set for FreeBSD: >>>=20 >>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-= init >>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebs= d >>>=20 >>>=20 >> Your first link is a 404, but this looks really nice! >=20 > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to > 'jet9-multitask-flow' to avoid confusion with naming, because it's not > as it seems, from name, an init replacement, but just a parallel task > launcher. And now the actual repository URL is > https://github.com/cvss/jet9-multitask-flow >=20 > In this commit I've complete the FreeBSD compatibility. Now a script > or dependency name can include minuses `-` and dots `.` (the first > stone I stumbled over was ftp-proxy). And I've cleaned up the main > script code > https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and > have split it to more functions that can be redefined. So it's now > easier to rewrite dependency extraction for FreeBSD rc-scripts - > current implementation is too rough and takes three 'awk' runs for > each rc-script. The last is not only time loss, but as DMarck > mentioned before, using awk restricts parallel rc to be run only after > FILESYSTEMS stage is done. Maybe it would be better to add to the > rcorder(8) some new option to dump the gathered dependencies in > tsort-compatible listing and insert them directly to flow execution > plan. >=20 > I've also added rc.conf variable `rc_parallel` to turn on and off > parallel execution. There's a risk to discover an incomplete > dependency annotations in some rc-scripts that earlier were masked by > serial execution. I've done some checks by enabling as much rc.conf > variables as possible to start more rc-scripts, and didn't found any > error. But it looks too good to be true and I'm afraid that it's just > to poor testing. I think if some ordering conflict will be found, it > could be worked-around with introducing a white-list for script names > that must be run only in sequentially. >=20 > So it remained first to check if it really works in different=20 > conditions. ------------------------------ Subject: Digest Footer _______________________________________________ freebsd-rc@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-rc To unsubscribe, send any mail to "freebsd-rc-unsubscribe@freebsd.org" ------------------------------ End of freebsd-rc Digest, Vol 470, Issue 1 ******************************************